NVIDIA’s New AI: The Biggest Leap In Robot Learning Yet

NVIDIA's DreamDojo is a new robot learning system that trains on 44,000 hours of human video (over 4 billion frames) to teach robots physical world understanding. It overcomes four key challenges: inferring actions from unlabeled video, compressing massive datasets to essential features, using relative rather than absolute coordinates for generalization, and preventing the AI from 'cheating' by feeding actions in small temporal blocks. A distillation step produces a student model running at ~10 fps — 4x faster than the teacher — while maintaining quality. The result is dramatically better physical simulation predictions (objects crumple, lids move correctly) compared to prior methods. All code and pre-trained models are released for free.