recordmydesktopのfaqには Recording 3d windows on the other hand, is not easy. You have to enable the full-shots option(*and with it you also definately have to enable usage of the MIT-Shm extension). This means though that now you are requesting a full image at every frame, so you will need a lot more proccessing power. In this case you should probably confine the recording only on the 3d window itself.