How Robots Are Really Trained: The Sim-to-Real Gap Is Not a Bug You Can Patch

Robot training is one of the most actively funded research areas in technology right now, and the sim-to-real gap — the stubborn failure of policies trained in simulation to hold up in the physical world — remains its central unsolved problem. This isn't a matter of needing faster GPUs or a more accurate physics engine. It is closer to an epistemological problem: the models we use to teach robots about the world are structurally different from the world itself, and no amount of rendering fidelity closes that kind of gap.

What Actually Breaks in Sim-to-Real Transfer#

Physics engines like MuJoCo, NVIDIA's Isaac Sim and Isaac Lab, and newer GPU-accelerated engines such as the open-source Newton are extraordinary engineering achievements. They are also deliberate approximations. Every simulation makes a contract with itself: objects have smooth, parameterized surfaces; friction is a scalar coefficient pulled from a lookup table; contact resolves cleanly at each timestep. The real world signs no such contract.

Consider friction alone. In simulation, the friction between a rubber gripper and a cardboard box is a number — usually a single Coulomb coefficient, sometimes a slightly richer parametric model, but a number either way. In reality, it depends on surface moisture, microscopic texture variation, the exact angle of approach, how long the surfaces have been in contact, and whether the cardboard has been handled before. None of those factors exist in the simulation's ontology. They are not poorly modeled — they are absent.

Contact dynamics are worse. The moment a robot finger touches an object, the physics become a chaotic small-scale interaction involving deformation, micro-slip, and force redistribution across irregular geometry. Physics engines resolve this with penalty-based or constraint-based approximations that are fast enough for training but diverge from measured reality at exactly the scales that matter for manipulation. The frontier has been moving here: 2026's purpose-built engines — Newton, a joint NVIDIA / Google DeepMind / Disney Research project, is the clearest example — add hydroelastic contact models, signed-distance-field collision, and deformable-body solvers specifically to make contact more faithful, and they run far faster than the previous generation. They genuinely help. But a better contact model is still a model. It makes the idealization richer and cheaper to sample, not real. A gripper that learned to pick up a cup in simulation still learned to pick up a mathematical idealization of a cup.

Sensor noise compounds all of this. Simulated depth cameras produce clean point clouds. Real depth cameras produce noise distributions that shift with lighting, surface reflectance, ambient infrared, and sensor temperature. Simulated IMUs don't drift. Real ones do, in ways that are partially predictable but never fully so. The robot trained in simulation is, in a meaningful sense, perceiving a different world than the one it gets deployed into.

Higher simulation fidelity helps at the margins. It does not close this gap, because the gap is not primarily about resolution. It is about the categorical difference between a closed formal system and an open physical one.

Imitation Learning's Built-In Contradiction#

Behavioral cloning — the simplest form of imitation learning — works by recording human demonstrations and training a policy to reproduce them. The appeal is obvious: humans already know how to do the task, so let the robot copy. The problem is that copying is not understanding, and the distinction bites the moment anything deviates from what was demonstrated.

This failure has a precise name in the literature: covariate shift, or compounding error. A cloned policy is only ever trained on the states a competent demonstrator visits. The first time the robot makes a small mistake, it lands in a slightly unfamiliar state, where its next action is a little worse, which lands it somewhere less familiar still. Errors compound instead of self-correcting, because nothing in the demonstrations taught the policy how to recover — skilled demonstrators rarely make the mistakes a learner most needs to see the fix for. Methods like DAgger patch this by iteratively asking an expert to label the states the policy actually visits, but that quietly reintroduces the human-in-the-loop cost imitation was supposed to eliminate.

The deeper structural problem is that imitation learning needs demonstrators who are simultaneously consistent and diverse. Consistent, because the learning signal degrades badly when different demonstrators solve the same subtask with incompatible strategies — the model gets a contradictory gradient and learns a blurry average of approaches that may work for none of them. Diverse, because a policy trained only on consistent demonstrations from one operator in one setting will fail catastrophically when the lighting changes, the object is slightly different, or the starting configuration is unfamiliar.

These two requirements fight each other. Real human demonstrators are not consistent — they adapt their strategy fluidly to context, and that adaptation is invisible to a system that only observes motor outputs. Two demonstrations of the same pick-and-place task from the same person can differ in ways that are irrelevant (stylistic variation) or deeply relevant (genuine strategic adaptation), and behavioral cloning has no mechanism to tell the two apart.

Approaches that add reinforcement learning feedback on top of imitation — sometimes loosely likened to RLHF in language models — help, because they give the policy a correction signal beyond mimicry. But the reward function now becomes the bottleneck. Specifying what "good" looks like for general manipulation is genuinely hard, and reward hacking — the policy maximizing the reward metric in ways that don't correspond to actually completing the task — is not a solved problem.

The critical point is that none of this is pattern matching on the way to comprehension. Behavioral cloning is statistical pattern matching, full stop. The policy has no model of the task's goal; it has a learned mapping from observations to actions that happened to earn reward on the training distribution. When that distribution shifts — and in the real world it always shifts — the policy has no principled fallback.

Domain Randomization: Useful Patch, Hard Ceiling#

The industry's primary response to the sim-to-real gap is domain randomization: deliberately inject variation into the simulation during training. Randomize friction coefficients across a range. Randomize lighting. Randomize object textures and masses. The intuition is that if the robot has seen enough variation in simulation, the real world will look like just another sample from that variation.

This works. Up to a point. Domain randomization has enabled real deployment for constrained tasks — locomotion over varied terrain, basic pick-and-place in controlled settings. The ceiling appears when the task demands precision beyond what randomization can cover, or when the real-world distribution falls outside the randomization envelope in ways nobody anticipated.

The more fundamental problem is that domain randomization is a patch applied to the wrong surface. It makes the policy more robust to variation within the simulation's ontology. It does not add new ontological categories. If the simulation has no concept of surface moisture affecting friction, randomizing the friction coefficient doesn't teach the robot to reason about moisture — it teaches the robot to be uncertain about friction, which is useful but categorically different.

There is also a cost. Policies trained with aggressive domain randomization are often less precise on any specific instance of a task than policies trained without it, because robustness and precision are in tension. The robot that can handle many friction values handles none of them optimally.

The Data Problem Nobody Advertises#

Imitation learning's scaling problem is severe and underreported. Training a manipulation policy that generalizes meaningfully — not just to held-out instances of the training distribution, but to genuinely novel objects and configurations — reportedly takes on the order of thousands of hours of demonstration data for non-trivial tasks. Some researchers put the number considerably higher for anything approaching household-scale generalization.

That is not a number compatible with a scalable deployment strategy built on human teleoperation. Collecting that volume costs serious time and money, introduces consistency problems as demonstrator fatigue and strategy drift accumulate, and still yields a policy whose real generalization radius is narrower than it looks on cherry-picked evaluations.

The current answer is to pool data across the whole field. Large cross-embodiment datasets — Open X-Embodiment, DROID, and the teleoperation fleets behind models like Physical Intelligence's π0 — exist precisely to attack the volume problem by sharing demonstrations across labs and robot types. It helps, and it is why today's models generalize further than the single-lab policies of a few years ago. But it doesn't dissolve the underlying issue, which is that "more data solves it" deserves scrutiny. More data of the same kind — human demonstrations of the same task family in the same environment — produces diminishing returns on generalization. What the data actually needs is not just volume but genuine distribution coverage of the deployment context, including failure modes, edge cases, and novel geometries. Collecting that is not a logistics problem; it is the problem of not knowing in advance what the deployment context will look like.

The Embodiment Gap: Hardware Is Not Interchangeable#

A policy trained on one robot body does not transfer trivially to a different robot body, even for the same task. This is the embodiment gap, and it is less discussed than sim-to-real but just as real.

The reason is that a learned policy is not an abstract procedure — it is a mapping from the specific sensorimotor space of the training hardware to motor commands for that hardware. Different joint configurations, actuator dynamics, sensor placements, mass distributions: each changes the input-output relationship in ways the policy was never trained to handle. A grasping policy that works well on one gripper may fail entirely on a gripper with different compliance, even when the geometric task is identical.

This has hard consequences for the industry's goal of general-purpose robots at scale. If every hardware revision requires substantial retraining, the cost of policy development scales with hardware iteration, not just task count. It is one reason the current wave of robot foundation models trains across many robot bodies at once. Physical Intelligence's π0 and NVIDIA's GR00T are explicitly cross-embodiment — one policy, many platforms — on the bet that a large pretrained model can absorb a new body with far less data than training from scratch. There is real evidence of positive transfer here. There is not yet evidence of "train once, deploy on any body": adapting to a genuinely new embodiment still takes meaningful task-specific data, just less of it than before.

What Foundation Models for Robotics Have Actually Shown#

Models like Physical Intelligence's π0, NVIDIA's GR00T, and Google DeepMind's Gemini Robotics — the successors to the RT-2 / OpenVLA line that opened the approach — represent a genuine architectural shift: using large pretrained vision-language models as the backbone for robot policies, on the bet that internet-scale pretraining encodes enough about physical relationships and task semantics to provide useful priors for robot learning.

The theoretical case is reasonable. The empirical case, examined honestly, is more constrained. In 2025 and 2026 these models moved out of tabletop demos and onto humanoid hardware — Figure's Helix, GR00T-powered platforms, Gemini Robotics driving commercial arms. That is a real milestone. Benchmark performance on controlled tabletop manipulation has improved meaningfully, and generalization to novel object categories shows genuine promise in the lab. What has not been demonstrated convincingly in peer-reviewed or rigorous third-party evaluation is reliable generalization across substantially different physical environments, tasks that require multi-step planning beyond a handful of steps, or recovery from failure modes not represented in training.

The benchmark-theater problem is acute here. A model that posts high success rates on a standard evaluation suite may be doing so by fitting the specific characteristics of that suite — camera placement, object set, lighting, table height — rather than demonstrating general physical competence. Evaluation methodology in robotics is not yet standardized in ways that make cross-paper comparisons trustworthy, which means a headline number and a deployable capability can be very different things.

Foundation models for robotics are neither hype nor solved. They are a promising research direction that has not yet demonstrated the kind of out-of-distribution generalization that would constitute closing the sim-to-real and embodiment gaps. Putting a VLA on a humanoid is not the same as reliability in an uncontrolled home or warehouse — and that reliability is the claim that matters and the one still unproven.

Where Honest Researchers Think This Goes#

Off the record, the researchers closest to this work tend to converge on the same view: the gap will narrow substantially but will not close on a short timeline, and the path to closure is not primarily more compute. The directions they flag most consistently are better grounding of policies in physical priors that are actually correct rather than approximately correct; denser real-world data-collection infrastructure that makes high-quality demonstrations cheaper to acquire; evaluation methodology that punishes overfitting to benchmark distributions; and architectures that can represent and reason about uncertainty in physical contact instead of averaging it away. The 2026 tooling wave — faster contact-rich engines like Newton, shared cross-embodiment datasets — maps onto exactly these bets, which is a good sign about the direction and a poor argument that the destination is close.

The timeline hedges are wide. Researchers who will say "five years" in a keynote will say "fifteen years, maybe" in a smaller room when asked specifically about household-scale generalization in uncontrolled environments.

The honest frontier position is that current methods work well for constrained, repeatable tasks in controlled environments, and that difficulty scales sharply with environment variability and task complexity. That is not a failure. It is a realistic map of the territory.

Key Takeaways#

• The sim-to-real gap is not a calibration problem; it is a structural mismatch between formal models and open physical systems. Better physics engines like Newton make the model richer, not real.
• Behavioral cloning fails through covariate shift: small errors take the robot into states it never saw demonstrated, and errors compound because nothing taught it to recover.
• Imitation learning produces statistical pattern matching, not task comprehension — the distinction matters the instant context shifts.
• Domain randomization is a useful robustness tool with a hard ceiling set by the simulation's ontological limits; it teaches uncertainty within known categories, not new categories.
• Demonstration-data requirements for genuine generalization are large enough to make imitation learning expensive at scale; cross-embodiment datasets help but don't dissolve the coverage problem.
• The embodiment gap couples policies to hardware. Cross-embodiment foundation models (π0, GR00T) show real positive transfer, but "train once, deploy on any body" is not here yet.
• Foundation models like π0, GR00T, and Gemini Robotics show real promise and now run on humanoids, but have not demonstrated the out-of-distribution reliability that benchmark scores can imply.
• More compute is not a self-sufficient answer. The data needs the right variation, the evaluation needs to measure real generalization, and the architecture needs to handle physical uncertainty rather than approximate it away.