Open Challenges in Real2Sim2Real
We believe that open discussion and deep collaboration are the best way to solve the hardest problems in any domain. This is an incomplete, evolving list of challenges we see in using simulation to enable real-world deployment at scale.
High-Fidelity World Grounding
Where does fidelity matter? Which gaps in perception, geometry, physics, or control cause deployment failures? How do we measure the fidelity a task actually needs?
Object Dynamics and Interactions. How can we reconstruct and simulate articulated objects, deformable materials, fluids, and their interactions? Which dynamics and material properties matter most for reliable transfer?
Tactile, force, and other input modalities. Which input modalities matter most for a task, and how do we simulate them with enough fidelity for reliable transfer?
Behavior Grounding: Learning Desired Behaviors
How can we shape simulated behaviors—from demonstrations, planning, or exploration—to match the behaviors we want in the real world?
Reconstructing demonstrations. How can we recover the motion, contact, and timing that matter—especially from egocentric human-object interactions with occluded contacts?
Data efficiency. Real-world data collection and reconstruction are expensive. What must we reconstruct, and how far can we extrapolate from a few demonstrations?
Recovery and diversity. How can we expand behavioral diversity and learn recovery behaviors while preserving the traits we want in deployment?
Going Beyond Human Performance
Beyond human embodiment. How can we teach quadrupeds and multi-arm systems behaviors that have no natural correspondence to human motion?
Superhuman behavior. How can simulation help robots exceed human precision, speed, or coordination—for example, in assembly tasks humans cannot reliably demonstrate?
Design for a Minimal Real2Sim2Real Gap
How can we design robots, physical assets and fixtures, sensors, and controllers to be easier to model and simulate? Which choices reduce the transfer gap while preserving the capabilities the task requires?
Closed-Loop Deployment Adaptation
Training the policy alone is not enough. Simulation must reflect the entire deployed system, from sensing and control to hardware and timing.
What does it take to translate success in simulation into reliable real-world performance? How can deployment feedback drive adaptation of both simulation and the system to close that final gap?