Chapter 7: From Humans to Robots: UMI, DexUMI, DEXOP, and EgoScale

Overview

Cheap human data becomes factory data only when embodiment gap, force observability, privacy, worker consent, and process IP are handled explicitly. This chapter rewrites the problem as a factory-cell data contract. The promise of large robot datasets in ^[1] and ^[2] matters, but in manufacturing an episode becomes valuable only when it is attached to inspection and process outcomes.

UMI is strongest for two-finger gripper tasks, DexUMI and DEXOP preserve more hand contact, and EgoScale-style egocentric video adds scale while still needing force and quality labels. The data is therefore not just a camera stream. It includes hand choice, force-torque, tactile patches, controller mode, operator intervention, and inspection result. ^[3] and ^[4] show why human-to-robot transfer loses critical signals unless the capture interface is designed around embodiment and contact.

After reading this chapter... - Explain factory manipulation as a data contract rather than a model score. - Separate two-finger, suction, custom, and five-finger choices by collection cost and failure observability. - Read papers, company releases, and deployment claims through evidence tiers. - Design the replay set and QA trace required for a first manufacturing PoC.

Core Map

Visual Argument

Figure 7.1. Human-to-robot transfer pathways. Source: reused local survey asset or author-created illustration.

Figure 7.2. DexUMI converts human hand demonstrations to robot hands. Source: reused local survey asset or author-created illustration.

Figure 7.3. DEXOP captures direct-contact dexterous demonstrations. Source: reused local survey asset or author-created illustration.

Three Losses in Human Data

The practical meaning of three losses in human data is that process variables must be left in a learnable form. ^[1] shows how scale can broaden embodiment and task families, yet a factory cell is narrower and stricter. A pick that looks identical to a person can become a different distribution when fixture tolerance, surface contamination, cycle pressure, or reject code changes.

In the concrete scenario, UMI is strongest for two-finger gripper tasks, DexUMI and DEXOP preserve more hand contact, and EgoScale-style egocentric video adds scale while still needing force and quality labels. Human demonstrations alone are not enough. Even when an interface such as ^[2] makes collection easier, a policy will repeat the same release failure if contact force and failure cause are absent. The episode schema must bind observation, action, contact state, QA outcome, and operator note under one key.

The evidence tier matters. A source such as ^[3] gives a method and benchmark that can be inspected. A company source can reveal deployment direction, but usually exposes less about data rights, recovery handling, and operating metrics. Manufacturers should place both in the same comparison table but never read them with the same confidence.

From a control perspective, the learned policy should not be the whole system. Force limits, guarded motion, fixture state, collision zones, and rollback conditions have to sit around the model. Without that boundary, collecting more data can create more safety stops and more rework instead of a more deployable robot.

From an operating perspective, the reproducibility of failure matters more than a headline success rate. A failure has to enter a replay set, the next update has to pass that replay set, and the deployed cell has to show a lower rate for the same failure code. The question is not only how to describe three losses in human data, but which missing log would make the claim collapse on a line.

UMI and Two-Finger Gripper Data

The practical meaning of umi and two-finger gripper data is that process variables must be left in a learnable form. ^[4] shows how scale can broaden embodiment and task families, yet a factory cell is narrower and stricter. A pick that looks identical to a person can become a different distribution when fixture tolerance, surface contamination, cycle pressure, or reject code changes.

In the concrete scenario, UMI is strongest for two-finger gripper tasks, DexUMI and DEXOP preserve more hand contact, and EgoScale-style egocentric video adds scale while still needing force and quality labels. Human demonstrations alone are not enough. Even when an interface such as ^[5] makes collection easier, a policy will repeat the same release failure if contact force and failure cause are absent. The episode schema must bind observation, action, contact state, QA outcome, and operator note under one key.

The evidence tier matters. A source such as ^[6] gives a method and benchmark that can be inspected. A company source can reveal deployment direction, but usually exposes less about data rights, recovery handling, and operating metrics. Manufacturers should place both in the same comparison table but never read them with the same confidence.

From a control perspective, the learned policy should not be the whole system. Force limits, guarded motion, fixture state, collision zones, and rollback conditions have to sit around the model. Without that boundary, collecting more data can create more safety stops and more rework instead of a more deployable robot.

From an operating perspective, the reproducibility of failure matters more than a headline success rate. A failure has to enter a replay set, the next update has to pass that replay set, and the deployed cell has to show a lower rate for the same failure code. The question is not only how to describe umi and two-finger gripper data, but which missing log would make the claim collapse on a line.

DexUMI, DEXOP, and Five-Finger Retargeting

The practical meaning of dexumi, dexop, and five-finger retargeting is that process variables must be left in a learnable form. ^[7] shows how scale can broaden embodiment and task families, yet a factory cell is narrower and stricter. A pick that looks identical to a person can become a different distribution when fixture tolerance, surface contamination, cycle pressure, or reject code changes.

In the concrete scenario, UMI is strongest for two-finger gripper tasks, DexUMI and DEXOP preserve more hand contact, and EgoScale-style egocentric video adds scale while still needing force and quality labels. Human demonstrations alone are not enough. Even when an interface such as ^[8] makes collection easier, a policy will repeat the same release failure if contact force and failure cause are absent. The episode schema must bind observation, action, contact state, QA outcome, and operator note under one key.

The evidence tier matters. A source such as ^[9] gives a method and benchmark that can be inspected. A company source can reveal deployment direction, but usually exposes less about data rights, recovery handling, and operating metrics. Manufacturers should place both in the same comparison table but never read them with the same confidence.

From a control perspective, the learned policy should not be the whole system. Force limits, guarded motion, fixture state, collision zones, and rollback conditions have to sit around the model. Without that boundary, collecting more data can create more safety stops and more rework instead of a more deployable robot.

From an operating perspective, the reproducibility of failure matters more than a headline success rate. A failure has to enter a replay set, the next update has to pass that replay set, and the deployed cell has to show a lower rate for the same failure code. The question is not only how to describe dexumi, dexop, and five-finger retargeting, but which missing log would make the claim collapse on a line.

Scale and Limits of Egocentric Video

The practical meaning of scale and limits of egocentric video is that process variables must be left in a learnable form. ^[10] shows how scale can broaden embodiment and task families, yet a factory cell is narrower and stricter. A pick that looks identical to a person can become a different distribution when fixture tolerance, surface contamination, cycle pressure, or reject code changes.

In the concrete scenario, UMI is strongest for two-finger gripper tasks, DexUMI and DEXOP preserve more hand contact, and EgoScale-style egocentric video adds scale while still needing force and quality labels. Human demonstrations alone are not enough. Even when an interface such as ^[11] makes collection easier, a policy will repeat the same release failure if contact force and failure cause are absent. The episode schema must bind observation, action, contact state, QA outcome, and operator note under one key.

The evidence tier matters. A source such as ^[12] gives a method and benchmark that can be inspected. A company source can reveal deployment direction, but usually exposes less about data rights, recovery handling, and operating metrics. Manufacturers should place both in the same comparison table but never read them with the same confidence.

From a control perspective, the learned policy should not be the whole system. Force limits, guarded motion, fixture state, collision zones, and rollback conditions have to sit around the model. Without that boundary, collecting more data can create more safety stops and more rework instead of a more deployable robot.

From an operating perspective, the reproducibility of failure matters more than a headline success rate. A failure has to enter a replay set, the next update has to pass that replay set, and the deployed cell has to show a lower rate for the same failure code. The question is not only how to describe scale and limits of egocentric video, but which missing log would make the claim collapse on a line.

Worker Consent and Process IP

The practical meaning of worker consent and process ip is that process variables must be left in a learnable form. ^[13] shows how scale can broaden embodiment and task families, yet a factory cell is narrower and stricter. A pick that looks identical to a person can become a different distribution when fixture tolerance, surface contamination, cycle pressure, or reject code changes.

In the concrete scenario, UMI is strongest for two-finger gripper tasks, DexUMI and DEXOP preserve more hand contact, and EgoScale-style egocentric video adds scale while still needing force and quality labels. Human demonstrations alone are not enough. Even when an interface such as ^[14] makes collection easier, a policy will repeat the same release failure if contact force and failure cause are absent. The episode schema must bind observation, action, contact state, QA outcome, and operator note under one key.

The evidence tier matters. A source such as ^[15] gives a method and benchmark that can be inspected. A company source can reveal deployment direction, but usually exposes less about data rights, recovery handling, and operating metrics. Manufacturers should place both in the same comparison table but never read them with the same confidence.

From a control perspective, the learned policy should not be the whole system. Force limits, guarded motion, fixture state, collision zones, and rollback conditions have to sit around the model. Without that boundary, collecting more data can create more safety stops and more rework instead of a more deployable robot.

From an operating perspective, the reproducibility of failure matters more than a headline success rate. A failure has to enter a replay set, the next update has to pass that replay set, and the deployed cell has to show a lower rate for the same failure code. The question is not only how to describe worker consent and process ip, but which missing log would make the claim collapse on a line.

Manufacturing Cell Checkpoint

Start a PoC by writing the data contract before selecting the model. UMI is strongest for two-finger gripper tasks, DexUMI and DEXOP preserve more hand contact, and EgoScale-style egocentric video adds scale while still needing force and quality labels. The contract should include episode ID, operator or teleop ID, hand or tool ID, contact channel, controller mode, quality decision, defect code, override reason, and rollback condition. If a vendor hides or cannot export these fields, the manufacturer cannot explain why model improvement happens.

The second checkpoint is evidence tiering. Papers with arXiv or DOI links support method claims; official company pages support product-direction claims; press and watchlist items should not become load-bearing claims. PI, Generalist, Skild, Figure, Covariant, Dexterity, Chef Robotics, Sunday, Config, and CarbonSix all fit under data-driven manipulation, but they expose very different evidence about manufacturing readiness.

Open Questions and Failure Modes

First, large data without contact state leaves insertion, wiping, deformable handling, and tool use under-observed. Second, when data rights stay entirely with the vendor, the manufacturer does not own the cause of process improvement. Third, worker video and process IP are learning assets and governance risks at the same time. Fourth, sim-to-real data does not guarantee factory performance unless it is tied to real QA labels.

Data Contract Addendum

The important unit is not the model name but the observable attempt. To a human, two factory cycles can look like the same task; to a robot, they may differ in surface friction, initial pose, contact order, force limit, and inspection rule. Data strategy is therefore an operating system for connecting failures to process variables, not a campaign for collecting success clips.

Factory data is slower and messier than benchmark data, but it carries more decisive labels. Defect codes, rework, operator intervention, line stops, and safety stops are not noise to be hidden from learning. They are the labels that decide whether a policy can be deployed. If those labels are detached from episodes, model curves can improve while process KPIs stay flat.

Large-data driven manipulation is therefore not a choice of one VLA. It is a joint design of task schema, hand choice, controller boundary, replay set, QA trace, and update governance. Without that design, a foundation model may produce impressive demonstrations without becoming repeatable replacement labor in manufacturing.

Operating Loop Addendum

The important unit is not the model name but the observable attempt. To a human, two factory cycles can look like the same task; to a robot, they may differ in surface friction, initial pose, contact order, force limit, and inspection rule. Data strategy is therefore an operating system for connecting failures to process variables, not a campaign for collecting success clips.

Factory data is slower and messier than benchmark data, but it carries more decisive labels. Defect codes, rework, operator intervention, line stops, and safety stops are not noise to be hidden from learning. They are the labels that decide whether a policy can be deployed. If those labels are detached from episodes, model curves can improve while process KPIs stay flat.

Large-data driven manipulation is therefore not a choice of one VLA. It is a joint design of task schema, hand choice, controller boundary, replay set, QA trace, and update governance. Without that design, a foundation model may produce impressive demonstrations without becoming repeatable replacement labor in manufacturing.

Deployment Governance Addendum

The important unit is not the model name but the observable attempt. To a human, two factory cycles can look like the same task; to a robot, they may differ in surface friction, initial pose, contact order, force limit, and inspection rule. Data strategy is therefore an operating system for connecting failures to process variables, not a campaign for collecting success clips.

Factory data is slower and messier than benchmark data, but it carries more decisive labels. Defect codes, rework, operator intervention, line stops, and safety stops are not noise to be hidden from learning. They are the labels that decide whether a policy can be deployed. If those labels are detached from episodes, model curves can improve while process KPIs stay flat.

Large-data driven manipulation is therefore not a choice of one VLA. It is a joint design of task schema, hand choice, controller boundary, replay set, QA trace, and update governance. Without that design, a foundation model may produce impressive demonstrations without becoming repeatable replacement labor in manufacturing.

Field Failure Addendum

The important unit is not the model name but the observable attempt. To a human, two factory cycles can look like the same task; to a robot, they may differ in surface friction, initial pose, contact order, force limit, and inspection rule. Data strategy is therefore an operating system for connecting failures to process variables, not a campaign for collecting success clips.

Factory data is slower and messier than benchmark data, but it carries more decisive labels. Defect codes, rework, operator intervention, line stops, and safety stops are not noise to be hidden from learning. They are the labels that decide whether a policy can be deployed. If those labels are detached from episodes, model curves can improve while process KPIs stay flat.

Large-data driven manipulation is therefore not a choice of one VLA. It is a joint design of task schema, hand choice, controller boundary, replay set, QA trace, and update governance. Without that design, a foundation model may produce impressive demonstrations without becoming repeatable replacement labor in manufacturing.

What to Learn Next

The next chapter applies the same principle to hands and end-effectors. Some cells are best served by suction or two fingers; others cannot close the data loop without five-finger hands and tactile or force-rich data.

References

1. Ha, Huy (2024). Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. arXiv.
2. Choi, Hojung (2026). In-the-Wild Compliant Manipulation with UMI-FT. arXiv.
3. Xu, Mengda (2025). DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation. arXiv.
4. Fang, Hao-Shu (2025). DEXOP: Passive Exoskeleton for Direct-contact Dexterous Demonstration. arXiv.
5. Si, Zilin (2025). ExoStart: From 10 Exoskeleton Demos to Dexterous Robot Manipulation. arXiv.
6. Qin, Yuzhe (2023). AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System. arXiv.
7. Ding, Zihan (2024). Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning. arXiv.
8. Kareer, Simar (2024). EgoMimic: Scaling Imitation Learning via Egocentric Video. arXiv.
9. Zheng, Renhao (2026). EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data. NVIDIA Research.
10. Yang, Fengyu (2023). Touch and Go: Learning from Human-Collected Vision and Touch. arXiv.
11. Feng, Ruoxuan (2025). AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-Tactile Sensors. arXiv.
12. Choi, Hojung (2025). CoinFT: A Coin-Sized, Capacitive 6-Axis Force Torque Sensor for Robotic Applications. arXiv.
13. Yu, Wenhao (2025). ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation. arXiv.
14. Huang, Yuhang (2025). Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization. arXiv.
15. O'Neill, Abby (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv.
16. Khazatsky, Alexander (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. arXiv.
17. AgiBot World Team (2025). AgiBot World Colosseo: A Large-Scale Manipulation Platform. arXiv.
18. Shaw, Kenneth (2023). LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning. arXiv.
19. Lambeta, Mike (2024). Digitizing Touch with an Artificial Multimodal Fingertip. arXiv.
20. Figure AI (2026). Figure 03 + Helix 02: General-Purpose Humanoid System. Company product page.
21. Physical Intelligence (2025). OpenPI: Open Source Robot Policy Stack. GitHub.
22. Black, Kevin (2024). pi0: A Vision-Language-Action Flow Model for General Robot Control. arXiv.
23. Octo Model Team (2024). Octo: An Open-Source Generalist Robot Policy. arXiv.
24. Mandlekar, Ajay (2021). What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. arXiv.