Chapter 4: Contact Models — Foundations for Control and Transfer

Overview

Contact is not merely a sensor event. A few millimeters of trajectory error may be invisible in free space, yet the same error becomes a jam, scratch, overload, or tool slip during insertion, press fitting, wiping, fastening, or cable routing. Scale alone does not erase this problem. Unseen material properties, tolerances, wear, sensing delays, and controller dynamics meet in a single contact, separating average policy performance from the tail of real defects.

This chapter argues that contact models and simulators do not save data by duplicating reality. They save risky exploration, expand neighborhoods around known failures, and reject weak policy candidates before hardware exposure. That role becomes release-relevant only when trajectory, contact, timing, and failure-ranking errors are measured against held-out real replay. The decisive question is therefore not which engine looks most realistic, but which errors were calibrated and which deployment decisions those errors are allowed to influence.

After reading this chapter... - Trace the lineage from impedance control and contact optimization to data-efficient learned control. - Distinguish system identification, domain randomization, and residual learning as complementary error-reduction tools. - Separate OpenUSD and Omniverse authoring, Isaac Sim execution, Isaac Lab training, Newton physics, and Cosmos augmentation. - Distinguish simulation correlation from physical fidelity and vendor capability evidence from independent transfer evidence. - Design a release loop in which real replay, calibration, safety, and quality traces remain authoritative.

Historical Lineage: Design the Interaction, Not Just the Trajectory

Impedance control established the premise that commanding position alone is inadequate when a robot couples to an environment. Hogan treated the dynamic relation between force and motion as the design object ^[25]. Khatib's operational-space formulation organized motion and force control around task-space dynamics ^[26]. The controller was no longer just a trajectory follower: it specified where the robot should yield, how strongly it should resist, and when interaction should stop.

Contact planning extended that premise into trajectory generation. Contact-invariant optimization searched for behavior without requiring every contact transition to be enumerated beforehand ^[27]. Direct trajectory optimization through rigid contact placed contact dynamics inside the optimization problem ^[28], while MuJoCo made fast contact simulation practical for iterative model-based control and learning experiments ^[29]. These methods reduced search, but nonsmooth contact, approximate friction, inaccurate geometry, and sensitivity to initialization remained.

The lineage matters because it separates policy skill from low-level stability. A visual policy may propose an approach pose, an estimator may detect the contact transition, a compliant controller may govern in-contact motion, and an inspector may own completion. A large network can absorb these functions, but deployment responsibility still decomposes at the same boundaries. Contact models are therefore not historical baggage. They define the semantics of action labels and make failure attribution possible.

Contact Models Save Data

Contact-rich work has a large failure space. Peg insertion, gasket seating, wiping, press-fit, and screw start can fail because of millimeters of pose error or small changes in friction. Covering that space with pure trial-and-error would create too many real factory failures. Contact models and controller constraints reduce the space that must be explored and exclude dangerous regions before learning begins.

Figure 4.1. An MPC receding horizon computes future cost from the current state and executes only the first control. Using it as a manufacturing guardrail requires contact, force, and rollback constraints to be modeled separately. Source: illustration by author (Gemini assisted).

This view does not reject data scale. QT-Opt and large-scale grasping showed that repeated collection can be powerful for vision-based manipulation ^{[5] [7]}, while Diffusion Policy and ACT-style work showed the value of action distributions and temporal chunking for complex manipulation ^{[8] [9]}. The manufacturing point is that a policy action should be wrapped by physical boundaries. Without force limits, guarded moves, impedance settings, collision zones, and fixture state, a strong benchmark policy can still create line stops.

Write the Control Boundary as a Table First

In a factory PoC, control design can be reviewed as a table before it becomes code. The table below separates what the learned policy may learn from what deterministic layers must prevent.

Without this table, the data flywheel becomes hard to interpret. A failure may be a policy error, a missing controller guard, or a fixture-model mismatch. Offline demonstration work can clarify demonstration quality ^[10], but deployment needs the controller boundary behind the demonstration to be logged with equal care.

Sim-to-Real Is Not Closed by Randomization Alone

Domain randomization trains across a family of possible worlds instead of fitting one nominal simulator. Tobin et al. randomized texture, lighting, camera, and object appearance so that real imagery could look like another training variation ^[30]. Peng et al. randomized masses, friction, damping, and related dynamics for control transfer ^[31]. The technique is useful when every parameter cannot be identified exactly, especially when parallel simulation can expose a policy to rare combinations.

Domain randomization improved transfer in the reported embodiments and tasks, but widening randomization is not a substitute for system identification or sim-real correlation testing. This boundary is supported by the reported transfer studies ^{[33] [31] [30]}. Visual randomization does not establish contact-force transfer, while dynamics randomization does not prove that the real structured error lies inside the training distribution.

Domain randomization is useful but insufficient. The randomized parameters must map to real defect modes. In an insertion cell, varying friction, clearance, and initial pose is valuable. But unless those variations are connected to real jam codes, scratch codes, and operator recoveries, simulation creates many scenes without creating release confidence.

Figure 4.2. The diagram places a real parameter point inside a randomized gravity, friction, and link-mass space. Inclusion in this illustrative space does not establish real-replay or QA validity. Source: illustration by author (Gemini assisted).

Simulation evaluation research offers ways to test real-world policies in simulation ^[1], but a manufacturer should not treat simulation results as deployment permission. Simulation is stronger at rejecting bad release candidates. A policy that cannot reproduce known failures, frequently violates force limits, or becomes unstable under fixture variation can be eliminated before the real cell. A surviving policy still needs limited real rollout and QA gates.

Randomization can also be too broad or structurally wrong. A policy trained over independent extreme values may become slow and conservative. Worse, it may see combinations that never occur while missing correlations that do: temperature with friction, wear with pose bias, or exposure with specular appearance. Real manufacturing data should therefore identify not only parameter bounds but also their correlations.

Chebotar et al. closed the loop by updating a simulation-parameter distribution from discrepancies in real trajectories ^[32]. The method uses reality more directly than fixed randomization, yet it still requires safe hardware exposure and cannot identify contact modes absent from the simulator. The practical sequence is identification of observable effects, structured randomization of the residual uncertainty, and held-out correlation tests for policy and failure ranking.

Decision Walkthrough: Press-Fit Insertion

Consider a press-fit insertion cell. The policy can use camera input and an instruction to bring the part above the fixture. The moment contact begins, however, the physical boundary becomes more important. Clearance is small, surface friction changes, and a slight tilt can create a force spike. If the policy keeps pushing, a good-looking action becomes a scratch or jam.

The first boundary is approach. The policy may propose a pose, but deterministic monitors should block no-go zones, fixture collision, and human entry. The second boundary is contact onset. If the force/torque trace leaves the expected envelope, insertion should stop and move to a recovery pose. The third boundary is completion. Even if the policy predicts success, the system should not mark completion unless inspection and force signatures agree. These boundaries are what let Diffusion Policy or ACT-style action modeling be used safely in a factory ^{[8] [9]}.

In this walkthrough, simulation is not a tool for making the policy more aggressive. It is a tool for rejecting unsafe candidates around known jams, fixture offsets, and friction changes. Repeated learning from QT-Opt and large-scale grasping remains important ^{[5] [7]}, but press-fit work cannot afford to create many real failures on the line. The contact model saves that failure budget.

Turning Boundary Evidence Into Release Gates

A control boundary is not useful just because it appears in a document. Each boundary needs evidence that can enter the release gate. The table below lists minimum gates for a contact-rich cell before a policy update is allowed.

The point is not to distrust learned policies. It is to separate what they can own from what deterministic systems must own. A manufacturer should not use model score as the release gate directly. It should ask what evidence the policy output left inside the physical boundary.

Drake-Style Thinking Separates Ownership

The value of Drake-style thinking is that the robot is not treated as one learned black box. The plant model, contact model, optimizer, controller, estimator, policy, and monitor have different responsibilities. A manufacturing cell needs the same separation. A vendor may supply the policy, but the manufacturer should own fixture geometry, safety envelope, quality gate, and rollback authority.

This separation matters when working with vendors. Isaac Lab, GR00T, Gemini Robotics, pi0, and Octo-style generalist policy work can provide broad priors ^{[19] [18] [20] [21] [22]}. The factory question is not only what the policy knows. It is what stops the robot when the policy does not know: unknown contact, sensor dropout, unmodeled fixture change, human entry, or inspection mismatch.

System Identification: Reality Supplies the Numbers

A simulator consumes geometry, mass, inertia, friction, elasticity, damping, actuator response, sensor delay, and controller timing. Design files supply only some of them. Production tolerances, residue, cable history, gearbox friction, force-sensor bias, and controller implementation vary in the cell. System identification is the experiment that connects those quantities to real input-output behavior. It is not a one-time ritual; it is versioned by embodiment, tool, material lot, sensor, controller, and software release.

A more expressive contact or deformable solver does not establish accurate real behavior without identified geometry, material, actuator, sensor, and controller parameters. The nonconvexity and initialization sensitivity of classical contact optimization already expose this boundary ^{[28] [27]}. NVIDIA's Newton material discloses broader contact and deformable capabilities, but it is first-party capability evidence, not proof that unknown cell parameters are recovered automatically ^[43].

Identification does not require recovering every physical constant. It requires the smallest effective model that distinguishes release-relevant failure. In a press fit, friction and angular misalignment may produce similar force signatures, making them unidentifiable from one trajectory. Additional excitation, speeds, or sensors may be necessary. Conversely, fitting parameters that cannot affect a release decision merely adds cost and non-identifiability. The target is not a visually impressive twin; it is a model that ranks failures and predicts guard activation within a measured envelope.

Calibration, policy training, and held-out replay must use separate episodes. Evaluating on the same trajectories used to fit the simulator hides overfitting and shared blind spots. Maintenance, tool replacement, controller gain changes, and sensor recalibration should open a new lineage. Otherwise the simulator and policy can memorize the same cases and appear to validate one another.

Residual Learning: Preserve the Model and Learn Its Error

Residual learning sits between analytic control and end-to-end policy learning. An explicit model or established controller produces a nominal action, while a learned component predicts a dynamics residual, observation correction, or bounded action adjustment. This can reduce the learned search space and retain interpretable guardrails. Yet a large residual effectively overwrites the base model, so saturation, magnitude, and intervention must be monitored.

ASAP illustrates this pattern for agile humanoid motion. It first trains a motion policy in simulation, uses roughly twenty minutes of real-robot rollouts to learn a delta-action model of the sim-real discrepancy, and then fine-tunes in simulation augmented by that model ^[34]. The result is strong evidence that small real datasets can correct a useful simulator in the reported Unitree G1 motion tasks. It is not evidence of automatic transfer to tight-tolerance insertion: the residual is embodiment-specific, and the reported tasks emphasize whole-body motion rather than manipulation contact.

Residual placement changes what a result means. A dynamics residual corrects state transition, an action residual modifies the controller command, and an observation residual corrects sensing. Combining all three may improve success while erasing attribution. A production trace should therefore log residual magnitude, saturation, base-controller intervention, sensor state, and quality outcome. A sudden residual increase after maintenance may be process drift rather than policy regression.

Calibrated Explicit Simulation and Policy Evaluation

Contact validity should be decomposed into at least four error budgets. Trajectory error covers the robot and object motion. Contact error covers force, contact timing, slip, and tactile state. Temporal error covers sensing, compute, communication, and stopping latency. Ranking error asks whether the simulator orders policies and failure conditions as reality does. Pixel similarity is neither necessary nor sufficient for all four.

SIMPLER aligns visual and control distributions and measures correspondence between simulated and real policy outcomes ^[1]. Its important lesson is that a simulator can become a useful evaluation proxy without being a perfect twin. Its equally important limit is that correlation is local to the policy family, task, embodiment, and shifts tested. A useful average ranking does not validate hidden contact force, part damage, or the safety of an individual episode.

Simulation is fit for a release-relevant role only after task-relevant trajectory, contact, timing, and failure-ranking errors are measured against held-out real replay. The cited reconstruction, policy-ranking, and adaptation studies support this bounded role ^{[35] [1] [32]}. This boundary does not authorize simulation to approve deployment alone; it specifies which candidates or regressions the calibrated proxy is allowed to reject.

SimFoundry represents a frontier in automated scene reconstruction and policy evaluation, but its disclosed limitations include upstream model nondeterminism, a single-support-plane assumption, reconstruction time and artifacts, and tabletop-focused evidence ^[35]. An automatically generated twin is therefore a draft evaluation asset. Geometry, articulation, collisions, physics properties, sensor placement, units, and coordinate frames still require review and real replay.

NVIDIA's Physical-AI Strategy as a Responsibility Chain

An NVIDIA-centered strategy is most accurate when described as a layered responsibility chain, not a claim that Cosmos replaces explicit simulation or real data. Authoring and interchange, sensor and physics execution, batched training, richer contact solving, and learned augmentation address different gaps. Real replay, calibration, safety, and quality traces remain authoritative.

OpenUSD supplies composable scene description and interchange; it is not a physics-fidelity certificate ^[44]. Omniverse libraries and connectors let existing content tools attach physics properties, semantics, sensor settings, and reusable USD structure ^[45]. The user-supplied Blender demonstration shows ovphysx, ovrtx, and validation tooling inside an authoring workflow ^[46]. That official video establishes the disclosed capability and version direction. It does not independently establish sim-real transfer, correct agent-filled properties, or perfect fidelity.

Isaac Sim executes the authored scene, robot, supported physics, rendering, and sensors ^[40]. Isaac Lab organizes parallel environments, reinforcement or imitation learning, curricula, randomization, and evaluation above that execution layer ^[41]. TacEx demonstrates one effort to simulate GelSight-style tactile observations in Isaac Sim ^[42], but implementing a tactile sensor model does not by itself validate real contact patches or lifecycle drift.

Isaac Sim supplies scene, robot, sensor, and supported-physics execution, whereas Isaac Lab organizes batched learning and evaluation; Newton expands contact and deformable options, but none of them alone is a safety case. The component roles are supported by the cited survey, framework paper, and first-party release ^{[40] [41] [43]}. Gao et al. ^[40] is a third-party survey of Isaac Sim, not official NVIDIA platform evidence. First-party documentation establishes disclosed interfaces and configurations, not independent factory-scale validity.

Newton belongs at the physics layer rather than the authoring or policy layer. NVIDIA positions it as an extensible, GPU-oriented engine for contact-rich manipulation, deformables, and integration with OpenUSD and learning frameworks ^[43]. Greater expressiveness can make cable bending, cloth, volumetric materials, and complex collision representable. Accuracy still depends on identified cable stiffness, connector tolerance, friction, actuation, sensor response, and controller timing. Vendor and partner demonstrations show disclosed capability; they do not prove long-term transfer across factories.

Cosmos and Learned Predictive Models

Learned predictive control predates today's foundation models. Visual Foresight combined action-conditioned video prediction with sampling-based planning toward image goals ^[36]. It showed that predicted pixels could guide robot pushing, while also exposing long-horizon blur, planning cost, and the gap between pushing and precision insertion. A plausible video is not evidence that contact force is correct.

DreamDojo combines large-scale egocentric human video with robot post-training, while World Action Models are Zero-shot Policies studies joint generation of future state and action ^{[37] [38]}. These directions may exploit video without native robot action labels and widen behavioral coverage. Their limits include the hidden cost of large pretraining corpora, embodiment-specific adaptation, missing force and tactile variables, and action feasibility.

Cosmos 3 integrates generation, reasoning, and action-conditioned prediction within a physical-AI model family ^[39]. Official material describes forward dynamics from state and action, inverse dynamics from video, and a policy mode that produces an action with a future ^[47]. The supplied action-post-training tutorial demonstrates a specific data format and configuration, but supplies no independent task-success denominator, tactile recovery test, or general teleoperation-reduction benchmark ^[48]. A tutorial choice such as one hundred examples cannot be converted into a general sample-efficiency claim.

Learned world models can support augmentation or policy ranking only within demonstrated predictive validity and cannot be treated as exact contact simulators. The cited predictive-model studies support this bounded use ^{[38] [37] [36]}. Inverse action inference from video does not recover unobserved torque, contact pressure, tactile state, or safety margin.

The defensible teleoperation claim is amplification, not elimination. A small real seed may condition appearance and embodiment; learned models may then generate backgrounds, arrangements, futures, or candidate actions that explicit simulation and real replay filter. Human effort can shift from full demonstrations toward rare failures, corrections, and validation. Total data cost still includes world-model pretraining, embodiment post-training, synthetic filtering, physical calibration, and real-cell assurance.

Tactile and Force Evidence Strengthen the Physics Loop

Contact models do not stay reliable without sensing. In tasks where vision cannot directly observe contact state, tactile and force channels reveal model mismatch. DIGIT, AnyTouch, ForceVLA, and tactile-language-action work show touch and force moving into policy representations ^{[15] [14] [12] [13]}.

Those signals matter to controllers as well. If force spikes above threshold, a guarded stop should override continued policy action. If tactile patterns diverge from expected contact, insertion should pause and move into a recovery routine. If the hand or sensor changes, the replay set needs new tags. Low-cost dexterous hands such as LEAP Hand widen research access ^[16], but in manufacturing, calibration drift and maintenance events become part of the physics loop.

Data Reveals Where the Simulator Is Wrong

Physics-first thinking is not perfect. Friction coefficients, deformable materials, cable bending, and tactile contact patches are difficult to model. Even when Genesis-style engines and Isaac Lab-style frameworks increase scene diversity and rollout throughput ^{[4] [19]}, real part wear, residue, and fixture deformation can remain wrong. Simulation should therefore be read as a hypothesis generator, not as ground truth.

Field data tells the team where simulation fails. If QA records jams on a particular material lot, simulation should widen the stiffness or friction range around that lot. If real force traces spike earlier than the model predicts, the contact-onset model may be wrong. If tactile patterns diverge from expected contact, geometry or sensor calibration may be the issue. In a large-data strategy, data trains not only the policy but also the simulator and guard logic.

Simulation and Real-Cell Validation Are One Loop

The appeal of large synthetic rollout is obvious. Isaac Lab and GPU simulation can generate many trajectories quickly ^[19], and Genesis-style physics engines can increase asset and scene diversity ^[4]. In contact-rich manufacturing, however, synthetic throughput is risky when it is separated from real-cell validation.

Figure 4.3. Simulation and real-cell validation loop. Simulation imports real failures, expands them, and returns candidate policies to limited real-cell validation. Source: illustration by author (OpenAI assisted).

A good loop starts with real failure. When QA records a jam defect, the team extracts the pose, force, and fixture state around that defect. Simulation expands the neighborhood and rejects candidate policies. A surviving policy returns to limited real rollout, and the result updates both the replay set and simulation parameters. Synthetic data then amplifies real failures instead of pretending to replace reality.

Combining Generalist Policy With Cell-Specific Guards

GR00T, Gemini Robotics, pi0, and Octo-style generalist policy work provide broad priors ^{[18] [20] [21] [22]}. Those priors help with new instructions, object appearances, and motion families. They do not by themselves carry factory release responsibility. Without cell-specific guards, a generalist policy can act confidently in unknown states.

The deployed architecture is therefore layered. A high-level policy proposes candidate actions or subgoals. A mid-level controller manages contact mode and impedance. A safety monitor blocks human entry, collision zones, over-force, and sensor dropout. A QA gate decides whether the output part meets process criteria. The data flywheel updates policy datasets, guard thresholds, simulation parameters, and QA replay together.

Manufacturing Cell Checkpoint

For a first contact-rich PoC, write the control-boundary document before selecting the model architecture. The document should include allowed contact, force limits, guarded moves, emergency stops, operator handoff, quality inspection, and rollback triggers. Then attach the sensor and log that proves each item. A boundary without evidence is not an operational boundary.

The second checkpoint is evidence tiering. Physics and simulation papers provide method evidence; company developer pages reveal toolchain direction; production reports speak to operating readiness. Toyota Research Institute's large behavior model work points toward faster skill teaching ^[17], but factory release gates still have to be checked through defect codes, safety stops, rework, and downtime. AgiBot World is evidence for broad platform coverage, not automatic proof that a particular factory cell will pass QA ^[3].

The third checkpoint is a responsibility audit. Before release, decide who owns policy failure, model mismatch, controller bug, fixture wear, and operator procedure. Without that audit, a field failure can turn into a blame loop: the vendor calls it a data issue, the manufacturer calls it a model issue, and the integrator calls it a fixture issue. Drake-style separation is both a technical design and a responsibility design.

Release Drill: Practicing Failure on Purpose

A contact-rich cell cannot be validated only through normal operation. Before release, the team should deliberately reproduce known failures under controlled conditions. In an insertion cell, that can include fixture offset, high-friction lots, sensor dropout, operator handoff, and emergency stop. The goal is not to create many failures; it is to verify that boundaries fire in the intended order.

The drill should leave four records. First, what action the policy proposed in the unknown state. Second, how quickly the controller guard blocked or redirected it. Third, what message and recovery path the operator received. Fourth, what QA and replay records were written. These records must connect before the team can say the cell failed safely. It is not enough that the robot stopped.

The drill also connects simulation to real-cell validation. A jam reproduced in simulation should call the same guard during limited real rollout. A new failure discovered in the real cell should return to simulation as a stress case. Without this drill, synthetic throughput can create false confidence instead of release confidence.

The drill should be judged by explanation completeness, not cycle time. When a failure happens, the policy proposal, controller intervention, sensor evidence, QA result, and operator action should remain as one chain. If any link is missing, the same problem can return in the next update. In contact-rich tasks, "the robot stopped" is different from "the system can explain why it stopped."

This criterion translates the Tedrake lineage into manufacturing language. Model-based thinking does not end with a more elegant simulator. It records which states belong to the policy, which states belong to the controller, and which states are blocked by QA. A large-data strategy makes those responsibility boundaries easier to test and faster to revise.

In practice, those boundaries should appear in the release note. When a new model ships, the note should state which guard thresholds changed, which simulation stress cases were added, and which defect replays passed. Then the next failure can be traced to a boundary change rather than to a vague statement that "the model changed."

The physics in this chapter is not a conservative brake. It clarifies the region where the policy is allowed to learn, which improves data efficiency. Clear boundaries compress failures into smaller cause sets, and smaller cause sets make the next collection and simulation update faster. If cycle time becomes the blocker, the same boundary map shows where to relax constraints without removing the evidence needed for rollback.

The release audit should also keep a small library of denied actions. These are cases where the policy proposed a faster move, the controller or QA layer blocked it, and the block was later judged correct or incorrect. Denied actions are useful because they expose the exact border between learned behavior and engineered protection. They turn safety review from a yes/no ceremony into a source of training, simulation, and threshold updates.

Open Questions and Failure Modes

First, locomotion evidence does not transfer automatically to precision manipulation. Proprioceptive whole-body tasks often have a fixed embodiment and constrained contact structure. Manufacturing manipulation adds specularity, occlusion, tolerances, friction, deformation, and long-horizon recovery. ASAP and domain-randomized locomotion are important method evidence, not automatic evidence for insertion, cables, or flexible packaging ^{[34] [33]}.

Second, policy-ranking correlation is not safety equivalence. SIMPLER-style correlation is useful for ordering candidates, yet it does not certify hidden damage or force excursions in an individual episode ^[1]. Simulation should first reject candidates and prioritize expensive hardware tests.

Third, expressive physics increases calibration responsibility. Additional material models and contact terms can represent more phenomena while increasing non-identifiability. If the sensor suite cannot distinguish several parameter combinations, multiple plausible models may explain one trajectory. Model complexity should match the failure modes and observations the task actually supports.

Fourth, learned video plausibility is not physical validity. Long sequences may look natural while action and state diverge or contact becomes impossible. Cosmos 3's official material itself documents action-state inconsistency and implausible dynamics as failure modes ^[47]. Generated experience is a candidate for validation, not a shortcut around it.

Finally, independent long-duration factory-scale evidence remains limited. Official releases and preprints establish fast-changing capabilities within disclosed versions and tasks. They do not yet provide broad external replication across cells, parts, shifts, maintenance events, and safety cases. Production-readiness language should therefore be reserved for process-specific real evidence.

The conclusion is simple: physics and large data are not rivals. Physics makes data collection safer, and data reveals where the physics model is wrong. A practical manufacturing strategy does not choose one over the other; it makes the boundary between their responsibilities explicit.

Figure 4.4. Contact-model transfer gate from modeling to residual acceptance. This diagram is not a performance claim; it summarizes the operational chain that carries capability evidence through replay and quality decisions into release authority. Source: author-created SVG.

Further Reading and Bridge to Chapter 5

Force-informed action representation continues in Terry's DexForce paper note #3, the relation between human data and embodiment constraints in ExoStart paper note #9, and contact paths tied to outcome quality in RoboPaint paper note #15. These are interpretive notes; the evidentiary basis for this chapter remains the primary papers and clearly labeled official sources below.

This chapter defined the physical field on which a policy learns. Impedance control and contact planning fixed action semantics; identification and randomization addressed explicit-model error; residual learning and world models made remaining error and coverage learnable. NVIDIA's stack is most useful under the same division: authoring, execution, training, contact physics, and augmentation remain separate, while real replay remains authoritative.

Chapter 5 asks how imitation learning, reinforcement learning, and offline learning combine real demonstrations, synthetic experience, corrections, and failures inside these boundaries. The next question is no longer whether a simulator is sufficiently realistic in the abstract. It is which learning rule preserves control and quality boundaries while generalizing across the available evidence.

References

1. Li, Xingyu (2024). Evaluating Real-World Robot Manipulation Policies in Simulation. arXiv.
2. NVIDIA (2025). Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. NVIDIA Research.
3. AgiBot-World Contributors et al. (2025). AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems. arXiv.
4. Genesis Team (2024). Genesis: A Generative and Universal Physics Engine for Robotics and Beyond. Project page.
5. Kalashnikov, Dmitry (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv.
6. Dasari, Sudeep (2019). RoboNet: Large-Scale Multi-Robot Learning. arXiv.
7. Levine, Sergey (2016). Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. arXiv.
8. Chi, Cheng (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv.
9. Zhao, Tony Z. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. arXiv.
10. Mandlekar, Ajay (2021). What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. arXiv.
11. Choi, Hojung (2026). In-the-Wild Compliant Manipulation with UMI-FT. arXiv.
12. Yu, Wenhao (2025). ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation. arXiv.
13. Hao, Peng et al. (2025). TLA: Tactile-Language-Action Model for Contact-Rich Manipulation. arXiv.
14. Feng, Ruoxuan (2025). AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-Tactile Sensors. arXiv.
15. Lambeta, Mike (2020). DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation. arXiv.
16. Shaw, Kenneth (2023). LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning. arXiv.
17. Toyota Research Institute (2024). Large Behavior Models for Robot Manipulation. Company technical post.
18. Bjorck, Johan (2025). GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv.
19. NVIDIA (2025). Isaac GR00T N1 Open Foundation Model for Humanoid Robots. NVIDIA Developer.
20. DeepMind Robotics Team (2025). Gemini Robotics: Bringing AI into the Physical World. arXiv.
21. Black, Kevin (2024). pi0: A Vision-Language-Action Flow Model for General Robot Control. arXiv.
22. Octo Model Team (2024). Octo: An Open-Source Generalist Robot Policy. arXiv.
23. Khazatsky, Alexander (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. arXiv.
24. O'Neill, Abby (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv.
25. Hogan, Neville (1985). Impedance Control: An Approach to Manipulation. Journal of Dynamic Systems, Measurement, and Control.
26. Khatib, Oussama (1987). A Unified Approach for Motion and Force Control of Robot Manipulators: The Operational Space Formulation. IEEE Journal on Robotics and Automation.
27. Mordatch, Igor et al. (2012). Discovery of Complex Behaviors through Contact-Invariant Optimization. ACM Transactions on Graphics.
28. Posa, Michael et al. (2014). A Direct Method for Trajectory Optimization of Rigid Bodies Through Contact. The International Journal of Robotics Research.
29. Todorov, Emanuel et al. (2012). MuJoCo: A Physics Engine for Model-Based Control. IEEE/RSJ IROS.
30. Tobin, Josh et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IEEE/RSJ IROS.
31. Peng, Xue Bin et al. (2018). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. IEEE ICRA.
32. Chebotar, Yevgen et al. (2019). Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience. IEEE ICRA.
33. Siekmann, Jonah et al. (2021). Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning. RSS.
34. He, Tairan et al. (2025). ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills. RSS.
35. Ranawaka, Nadun et al. (2026). SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation. arXiv preprint.
36. Ebert, Frederik et al. (2018). Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv preprint.
37. Gao, Shenyuan et al. (2026a). DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos. ICML.
38. Ye, Seonghyeon et al. (2026). World Action Models are Zero-shot Policies. arXiv preprint.
39. NVIDIA Cosmos Team (2026). Cosmos 3: Omnimodal World Models for Physical AI. arXiv preprint.
40. Gao, Sicong et al. (2026b). NVIDIA Isaac Sim: Enabling Scalable, GPU-Accelerated Simulation for Robotics. arXiv:2606.03551. Third-party survey.
41. Mittal, Mayank et al. (2025). Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. arXiv preprint.
42. Nguyen, Duc Huy et al. (2024). TacEx: GelSight Tactile Simulation in Isaac Sim -- Combining Soft-Body and Visuotactile Simulators. arXiv preprint.
43. NVIDIA (2026a). Newton Physics Engine. Official technical documentation.
44. Alliance for OpenUSD (2026). OpenUSD Specifications and Alliance Governance. Official specification and governance site.
45. NVIDIA (2026b). Integrate Physical AI Capabilities into Existing Apps with NVIDIA Omniverse Libraries. Official technical post.
46. NVIDIA (2026c). Bringing Agent-Ready Simulation Into Blender. Official demonstration video.
47. NVIDIA (2026d). Develop Physical AI Reasoning, World, and Action Models with NVIDIA Cosmos 3. Official technical post.
48. NVIDIA (2026e). How to Post-Train NVIDIA Cosmos 3 for Robot Action Prediction. Official tutorial video.
49. Yuke Zhu et al. (2020). robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv preprint.
50. Rishabh Agarwal et al. (2021). Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS.
51. Ankur Handa et al. (2023). DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality. ICRA 2023.
52. Zilin Si et al. (2024). DiffTactile: A Physics-based Differentiable Tactile Simulator for Contact-Rich Robotic Manipulation. ICLR 2024.
53. Miquel Oller et al. (2024). Tactile-Driven Non-Prehensile Object Manipulation via Extrinsic Contact Mode Control. Robotics: Science and Systems (RSS) 2024.
54. Various (2025). Robust Model-Based In-Hand Manipulation with Integrated Real-Time Motion-Contact Planning and Tracking. arXiv preprint.
55. Uikyum Kim et al. (2021). Integrated Linkage-Driven Dexterous Anthropomorphic Robotic Hand. Nature Communications.