Chapter 5: Imitation and Reinforcement Learning — From Data to Policy

Overview

The research lineage connecting Abbeel, Levine, and Finn is not simply about collecting more demonstrations and fitting a policy. It developed guided policy search, behavior cloning, offline reinforcement learning, autonomous robot trials, simulation, and operator correction as different parts of one learning loop. ^[3] opened a path toward learning hand-eye grasping from large-scale real data. ^[2] showed how distributed real-robot grasp attempts can become assets for off-policy learning. ^[1] asks through controlled experiments which properties of offline human demonstrations actually determine manipulation performance.

In a factory cell, the central question is not the algorithm name. It is which experiences can be reused. Human demonstrations seed the initial distribution. Robot rollouts create hard negatives that people rarely demonstrate. Operator overrides capture the state just before failure and the corrective action. RL can improve search and recovery only when reward is tied to inspection, rework, scrap, and safety-stop outcomes; otherwise it optimizes a demo proxy rather than the process.

After reading this chapter... - Distinguish behavior cloning, interactive correction, offline RL, and online RL by data support, reset access, and risk. - Explain why covariate shift in behavior cloning differs from value overestimation in offline RL. - Count operator observation, intervention, reset, safety-stop, and revalidation costs in real-robot learning. - Explain why locomotion's sim-first successes do not directly validate contact-rich manufacturing manipulation. - Design reward, replay sets, quality traces, and rollback gates inside one manufacturing proof of concept.

Learning Modes in a Cell

The point of the table is practical. Human data can start the system, but manufacturing automation needs a cell that does not regress after updates. The data loop must therefore be an operating mechanism that moves replay, correction, and inspection outcomes together, not a campaign for collecting success clips.

Behavior cloning, offline RL, online RL, and intervention should be selected by reset access, exploration risk, data coverage, and release constraints. The studies by ^[1], ^[29], and ^[28] support this synthesis only within their reported task and data boundaries, not as a universal guarantee across embodiments or processes. The same update can belong to a permissive regime for damage-free tabletop picking and a tightly guarded regime for precision insertion that can create scrap.

The point is not to choose one regime forever. A team can initialize with behavior cloning, expand support through selective correction, and use online RL only for a bounded subskill. If safety stops or manual reset labor rises, it can close the online stage and return to the last approved offline policy. Choosing a learning algorithm is also choosing who may expose the cell to which states.

Behavior Cloning Fails Through State-Distribution Shift

Behavior cloning converts demonstration learning into supervised prediction of expert action a from demonstrated state s. Its simplicity is valuable, but deployment changes the input distribution. A small prediction error moves the robot into a state absent from the demonstrations; the next prediction has less support, so error can compound. Ross et al. ^[25] formalized this covariate-shift problem as online sequential prediction and introduced DAgger, which asks the expert to label states actually visited by the learner. The historical shift was from “collect more expert trajectories” to “label the states produced by the current policy.”

Interactive correction is not free supervision. A design that must visit a risky state to obtain its label cannot be copied directly into a cell where that visit can damage a product or fixture. Keeping an expert attentive at every timestep recreates much of teleoperation's observation cost. ThriftyDAgger gates control transfer by novelty and estimated completion risk, but it assumes a calibrated risk model and an available supervisor ^[32]. Evidence from one physical cable task and a small user study does not establish a general reduction in manufacturing labor.

A manufacturing implementation therefore needs three separate thresholds. An observation threshold flags an unfamiliar state. An intervention threshold determines when the operator receives control. A stop threshold uses hard force, speed, or collision guards when waiting for judgment is unsafe. The stored record must include the pre-intervention observation, autonomous proposal, correction, takeover delay, and final inspection outcome. Saving only the corrected action hides whether the loop actually reduced covariate shift or merely prevented visible crashes.

From Guided Policy Search to Scalable Real-Robot Learning

Guided policy search connected trajectory optimization or local controllers to high-dimensional neural policies, providing a route from structured teachers to visuomotor control ^[26]. Early contact-rich results relied on substantial instrumentation and task-specific setup, but they established a durable pattern: initialize from a competent teacher, then broaden the policy's state distribution. Large-scale grasping later extended that pattern by putting repeated attempts from multiple robots into replay and reusing them off-policy ^[3], ^[2]. The operational advance was not only a better loss; it was treating expensive robot experience as a reusable asset.

That lineage also exposes a reproducibility boundary. QT-Opt is strong evidence that real grasp outcomes and off-policy replay can form a learning engine, but its scope remains grasping-centric and its fleet and reset labor are difficult for a small manufacturer to reproduce ^[2]. MT-Opt and AW-Opt reuse demonstrations, heterogeneous offline logs, and online experience across related tasks, but the curated task graph and physical interaction cost remain ^[36], ^[37]. Scale does not independently establish transfer to arbitrary contact tasks or production readiness.

Demonstrations Start the System, but Do Not Own It

Demonstrations are valuable because skilled workers already know the insertion angle, the moment to relax force, and the wrist adjustment that clears a cable snag. That knowledge does not transfer directly into a robot action space. Camera placement, gripper compliance, finger count, velocity limits, and safety zones can turn a visually similar demonstration into a different distribution. ^[1] matters because it shows that data quality, coverage, and task structure can dominate the raw presence of human demonstrations.

Offline demonstration quality and coverage can matter as much as nominal episode count for manipulation policy performance. The reported evidence remains bounded to its tasks and datasets ^[27], ^[1]. An energy-based policy may represent multiple action modes, and a larger model may fit the log better, but neither guarantees calibrated decisions outside demonstration support. Recent self-improvement systems also change the distribution by adding outcome labels, autonomous experience, or expert corrections; scale alone does not fill unsupported contact states ^[45]. Counts such as “100 demonstrations” or “one million episodes” are therefore not comparable without sensors, failure ratio, task-stage coverage, operator diversity, and held-out split.

Figure 5.1. ALOHA's low-cost bimanual teleoperation hardware and example demonstration and learned tasks. The figure itself does not show the ACT policy architecture. Source: Zhao et al. (2023), arXiv:2304.13705, Figure 1.

Manufacturers should not let demonstrations become only raw videos owned by a vendor. Even if worker identity is protected, the episode should retain skill tier, tool ID, fixture revision, lot, pre-check result, cycle-time pressure, and final inspection label. Only then can a model update be interpreted as "the B-lot insertion jam decreased" rather than "success improved."

DAgger-Style Correction Is a Sensor Near Failure

The DAgger lesson is that people cannot provide every relevant state in advance. Some states appear only when the policy moves in the cell: a slightly twisted part, a contaminated suction cup, or a cable jacket caught on a fixture edge. In those moments, operator override is not nuisance data. It is the most expensive label.

A useful override log preserves three facts. It records the observation where the policy hesitated or chose the wrong approach. It records the corrective action. It records how that correction affected the final quality result. The cross-domain logic in ^[4] and ^[5] is important here. As robot and domain diversity grows, action conventions and embodiment gaps also grow; without correction logs, diversity does not automatically become factory robustness.

Intervention data is informative because it visits policy-induced weak states, but it remains operator- and safety-dependent. Runtime monitoring and HIL-SERL support that bounded interpretation in their reported settings ^[33], ^[32], ^[34]. Runtime forecasting can reduce later monitoring while still requiring full human monitoring for its warm-start round and remaining vulnerable to novel missed failures ^[33]. HIL-SERL combines demonstrations, a learned reward classifier, off-policy learning, and human intervention, but it does not remove competent early supervision, reward construction, or reset engineering ^[34]. A cost claim must count observation time, transition delay, reset, damage prevention, and revalidation, not only the number of corrected segments.

RL Belongs First in Repeatable Subskills

RL should enter the factory through repeatable subskills with clear episode boundaries and reward definitions: grasp candidate ranking, compliant insertion search, regrasp, and retreat-and-retry after a failed contact. ^[2] shows how large real grasp attempt logs and off-policy learning can form a robot data engine. Its strength also came from a narrow task scope: grasping.

Figure 5.2. Eight robotics environments benchmarked in Isaac Gym. GPU-parallel simulation expands rollout volume, but this figure alone does not establish transfer to a real manufacturing cell. Source: Makoviychuk et al. (2021), arXiv:2108.10470, Figure 1.

Simulation broadens this loop. Sim evaluation and GPU-accelerated training, as in ^[16] and ^[17], can turn rare failures and domain variation into regression tests. But if the simulator does not explain friction, compliance, and sensor latency well enough, the policy may become overly cautious or apply the wrong force in the real cell. A simulated rollout should therefore state which real failure it is trying to explain before it is treated as training data.

Offline RL Uses Logs but Cannot Know Values Outside Them

Behavior cloning follows logged actions. Offline RL instead tries to select actions with higher long-term reward from a fixed log. Its distinctive failure is that a value function can overestimate actions that barely occur in the dataset, after which the policy actively exploits the error. CQL adds a conservative regularizer that lowers values for unsupported actions ^[28]. MOPO penalizes model-generated transitions where a learned dynamics ensemble is uncertain ^[30]. Both define important boundaries, but conservatism can suppress rare useful recovery actions, and ensemble uncertainty can be uncalibrated when its models share the same bias.

The manufacturing question is not whether offline learning removes every online trial. It is which online trials can be avoided before approval. A candidate policy can be trained on fixed logs, evaluated on replay partitions separated by time, machine, and operator, then tested on hardware at reduced speed under action bounds. Hardware evaluation remains necessary because logs omit some sensor faults, wear, delay, and contact discontinuities. RL Unplugged separates offline algorithm comparison from environment access, but simulator logs do not reproduce operator intervention or hardware failure ^[31].

A learned Q-value is not a manufacturing-quality probability. It is expected cumulative reward under a defined reward and data-generating process, not a certificate of force-limit compliance or absence of defects. A policy can improve mean return while worsening a rare, expensive tail. Teams should therefore measure prohibited-action rate, force-limit violations, emergency stops, rework, and worst-case cycle time independently from the value estimate.

Three Offline-to-Online Operating Regimes

The first regime is offline initialization followed by bounded online fine-tuning. AWAC starts from offline data, imitates higher-advantage logged actions, and uses dynamic-programming values to continue online improvement ^[29]. This can reduce random initial exploration, but it does not remove physical resets or guarantee safe monotonic improvement. Manufacturing use should begin with a subskill whose action envelope, speed, and automatic reset are already controlled.

The second regime is human-in-the-loop reinforcement learning. The robot creates informative failures while a skilled supervisor takes control before they become product or equipment damage. Its advantage is direct exposure to the policy's weak states rather than repeated successful demonstrations. Its cost is sustained attention and reaction latency. If the operator is simultaneously responsible for production, “twenty interventions” can hide a much larger opportunity cost.

The third regime is shadow learning with batch approval. New data creates a candidate checkpoint rather than changing the production policy immediately. The candidate must pass frozen replay, simulation variations, a limited held-out hardware test, and a canary on a small number of cells. This slows online adaptation but restores auditability and rollback. In high-risk manufacturing, who approves an update and when is more important than the slogan that the policy learns continuously.

Sample Efficiency Is Not Operational Efficiency

SERL packages off-policy learning, demonstration reuse, reward computation, controllers, and reset mechanisms for real-robot training, making clear that system details determine observed sample efficiency ^[35]. DayDreamer learned pixel-based world-model policies with sparse reward across several physical robots ^[38]. These results weaken the claim that real-world RL is intrinsically impractical, but short controlled tasks do not validate factory shifts, wear, maintenance, or downstream quality tracing.

Real-world RL evidence must report resets, unsafe exploration controls, trial denominators, and the transition from training to guarded deployment. The cited systems provide evidence only within their reported supervision, task, and safety arrangements ^[34], ^[35], ^[39]. CPO constrains expected cost through a trust-region approximation; it is not a hard real-time safety barrier, and its simulation locomotion results do not establish contact-manipulation safety ^[39]. Safety PLCs, force limits, protective stops, and final human authority may need to remain independent of the learned objective.

Operational cost is wider than episode count. It includes clearing failed parts, returning fixtures to start state, replacing gripper pads and consumables, recalibrating cameras, releasing protective stops, inspecting damaged units, correcting reward labels, and revalidating candidates. An algorithm can halve robot trials yet lose operational efficiency if each reset takes twice as long or requires constant human observation. Report robot time, operator time, damage and consumable cost, inspection time, and safety events beside sample count.

Why Sim-First Locomotion Does Not Validate Contact-Rich Manipulation

Locomotion is the clearest domain in which sim-first learning has earned credibility. Policies trained across massively parallel environments with randomized mass, actuator gains, ground friction, terrain, and pushes have transferred to physical quadrupeds and bipeds ^[40], ^[41]. These policies often operate on a fixed morphology, consume joint and inertial history plus a motion command, and output joint targets. Their dominant contacts are concentrated at the feet and ground, a structure well suited to repeated simulation and parameter randomization.

Contact-rich manufacturing manipulation has a different uncertainty stack: camera occlusion and reflection, millimeter-scale gripper contact, friction, deformation of thin objects, tolerances, force and tactile response, inspection criteria, and multi-step recovery. Balance recovery does not measure a scratched insertion surface or damaged cable jacket. CPO's constrained simulation results and blind stair transfer are bounded locomotion evidence, not certification of assembly contact, quality, or safety ^[39], ^[41].

Simulation-generated experience may reduce some real trials, but final release still requires held-out real evaluation and rollback criteria. DexMimicGen and SimOpt support only bounded parts of that synthesis in their reported settings ^[42], ^[43]. Yet synthetic contact validity depends on calibrated assets and physics, and iterative correction cannot close a gap the simulator parameters cannot express. Held-out real evaluation must still check quality, safety, and cycle time against an approved rollback point. Simulation does not eliminate real validation; it helps decide where scarce real tests are most informative.

Decision Walkthrough: A Connector-Insertion Cell

Consider a cell that inserts a wire-harness connector into a plastic housing. The worker grasps the connector, orients the keyed side, rides over the housing lip, and confirms seating through a click or a force drop. A human demonstration shows the sequence, but the policy must learn finer pieces: approach pose, pre-insertion alignment, low-force probing, jam detection, retreat, retry, and final inspection.

Behavior cloning is suitable for the initial approach and wrist orientation. Operator correction teaches the retreat angle that avoids damage when the connector catches on the lip. Off-policy RL can be used for repeated subproblems such as probing depth or retry strategy. Simulation can vary housing tolerance, connector deformation, and fixture offset. The final reward, however, cannot be "pushed all the way in." It must combine seating inspection, force trace, cosmetic damage, and cycle time.

The walkthrough matters because data ownership is segmented. A manufacturer should not buy a vague bundle called "connector insertion data." The team should write which segment needs which signal. Approach pose may need video and joint state. Jam recovery needs force/torque and a failure code. Final seating needs inspection images and a quality label. Inside one episode, the right learning method and the necessary sensor can change several times.

Reward Design Must Be Translated Into Quality Language

The most common failure in applying RL to a factory cell is an easy surrogate reward. Rewarding proximity to the target or insertion depth can be useful in a benchmark, but it can be dangerous on a line. A deeper insertion reward can encourage over-force and damage. A cycle-time reward can encourage skipped inspection. A smoothness reward can hide a slow drift toward a bad contact path.

Manufacturing reward has to be translated into quality language. Success is not one pass/fail bit; it decomposes into reject code, rework, cosmetic grade, force-limit violation, operator override, and downstream test result. ^[2] is useful because each grasp attempt had an outcome. The manufacturing version needs a finer outcome label. If failures are numerous but not keyed by reason, an off-policy update cannot know what it is fixing.

Good reward design also avoids giving the policy every authority. Force limits and forbidden zones may be hard guards rather than reward terms. Cycle time is an optimization target, not a priority over safety stops. Human correction can shape reward, but the operator's skill tier and context should be retained. Otherwise the policy may confuse one worker's habit with a process rule.

Separate the Replay Board From the Release Board

The learning team's replay board and the production team's release board should be separate. The replay board collects failures densely: connector jam, mis-pick, part drop, soft collision, late cycle, operator takeover. Each bucket compares the old model with the candidate model. Seeing many failures is good at this stage because failure diversity exposes the update's weak spots.

The release board asks a different question: operational risk. A candidate can improve the replay set and still be rejected if the production canary increases cycle-time variance or safety stops. Company-facing large-behavior-model claims such as ^[18] often expose only part of this board. Manufacturers need their own raw release metrics to translate external claims into plant decisions.

Separating the boards clarifies the roles of imitation and RL. An imitation update may improve human-like smoothness and initial success. An RL update may improve rare-failure recovery. A simulation update may widen a tolerance range. But every update faces the same release questions. Did known failures decrease? Did new failures appear? If rollback is needed, which dataset and controller version does the cell return to?

Evidence Tiers: Reading Papers, Datasets, and Company Claims Together

The evidence in this lineage has different meanings. ^[3] provides a template for perception-to-action learning. ^[2] shows real robot attempts as a basis for off-policy RL. ^[4] and ^[5] ask how multi-robot and cross-domain data affect generalization. ^[7] and ^[8] point toward larger in-the-wild and cross-embodiment data.

Company technical posts and product pages provide a different kind of evidence: production intent and stack direction. They usually do not expose failure distributions or raw telemetry. Evidence tiering is therefore not a trust/no-trust choice; it asks what a source proves. Papers support methods and benchmarks. Dataset papers support coverage and schema. Company posts support deployment direction and integration claims. Factory readiness still requires plant-specific replay, canaries, and maintenance logs.

Without that distinction, large-data strategy is easily inflated. A broad public dataset does not imply that a factory's connector fixture is covered. A company claim about a large behavior model does not imply that the plant's scrap code has fallen. Manufacturers should map every external source into an internal evidence tier and define which decisions each tier permits.

Offline Data Gives Both a Common Language and an OOD Warning

RoboNet, Bridge Data, DROID, and Open X-Embodiment try to move beyond one lab's task by pooling robots and scenes ^[4], ^[5], ^[7], ^[8]. For manufacturers this gives two things. The first is pretrained representation and task prior. The second is an OOD warning. As public datasets become broader, plant-specific fixtures, cleaning state, tool wear, and quality criteria become more clearly local variables.

Architectures such as Diffusion Policy and ACT changed how action sequences are learned on top of such data ^[9], ^[10]. Diffusion represents multimodal action distributions more naturally; ACT chunks short-horizon bimanual sequences. But better action representation does not learn process failure if the dataset lacks reject codes and contact state.

Figure 5.3. Comparison of explicit, implicit energy-based, and diffusion policy representations. Diffusion Policy can represent multimodal action distributions, but it does not add quality labels absent from the data. Source: Chi et al. (2023), arXiv:2303.04137, Figure 1.

What the Manufacturer Should Keep Internally

Even if a vendor provides a strong policy stack, some layers should remain internal to the manufacturer. The first is task ontology. Names such as connector insertion, cable routing, and gasket placement are not enough. The manufacturer should own the segment taxonomy and failure codes: approach, align, probe, insert, inspect, recover. Without that vocabulary, replay sets disappear when the vendor or model family changes.

The second layer is the evaluation harness. Work such as ^[1] and ^[16] matters because it treats evaluation protocol as a durable asset, not merely a policy score. A factory cell needs the same principle. Candidate models should be compared against the same replay set, SKU split, inspection rule, and rollback condition. If the team sees only a vendor dashboard, it cannot tell whether the model improved or the metric changed.

The third layer is failure economics. Not every failure has the same cost. A mis-pick may be a cycle-time loss, while over-force can create scrap and downstream claims. Operator takeover reduces throughput but may be cheaper than product damage. If RL reward and release boards do not encode this cost difference, the policy may improve a high-count easy metric while leaving a low-frequency expensive failure untouched.

The final layer is a data export path. The manufacturer may not store every raw video internally, but episode index, features, labels, model decisions, failure codes, and override events should remain exportable. In large-data manipulation, the data moat is not only volume. It is the ability for the next supplier or internal team to replay past failures.

This principle matters even more for small manufacturers. They may not own a robot fleet or large RL infrastructure, but they can still own a task ontology and replay set. As a deliberately proposed starting heuristic rather than a published benchmark, the first replay set can be modest: 50 failure episodes and 10 gold success episodes may be enough to discipline a PoC. The key habit is that every update must face the same failures again. Once that habit exists, a vendor demo becomes a comparable experiment rather than a one-off presentation.

The same logic applies to staffing. The person who understands the process, the person who maintains the robot, and the person who evaluates quality should all be able to read the replay categories. If the replay taxonomy is intelligible only to ML engineers, it will not survive a shift handoff. If it is intelligible only to line engineers, it will not shape model training. The durable asset is the shared language between the two groups.

That shared language is what turns a pilot into an accumulating program.

Manufacturing Cell Checkpoint

The first PoC question is not whether to use behavior cloning or RL. It is which parts of the task start from demonstration, which parts are strengthened by operator correction, and which parts can safely gather hard negatives through off-policy RL or simulation. That decision comes from the task graph and safety boundary. Separate the segments where human hands are still better, where robots can generate repeated failures, and where simulation is trustworthy.

The second question is the replay set. A manufacturing replay set is not a bundle of benchmark videos. It is a bundle of quality failures: mis-pick, slip, over-force, insertion jam, cosmetic damage, late cycle, emergency stop. Each must be reloadable under the same keys. If a model update improves the replay set and then lowers the matching production failure code in a canary deployment, the loop is real.

The third question is evidence tier. Research such as ^[3], ^[2], and ^[1] supports method choices. Company technical posts such as ^[18] reveal deployment direction, but internal data and failure statistics are rarely fully exposed. Buy/build decisions should therefore track data rights, exportability, and rollback authority separately.

Open Questions and Failure Modes

First, large offline datasets still do not explain every contact-rich failure because many causes are site-specific. Second, if RL reward is disconnected from inspection, the policy optimizes a surrogate rather than quality. Third, if privacy constraints prevent operator correction from being stored in usable form, the most important edge-case labels disappear. Fourth, if a vendor separates model updates from raw episode lineage, the manufacturer cannot own the reason for improvement. Fifth, conservative offline learning can suppress rare recovery behavior while reducing extrapolation, and intervention learning can reduce supervision while creating new missed-risk paths. No regime can be compared by one success rate without its data support and safety system.

What to Learn Next

Chapter 6 reads the same data loop through policy architecture. The regimes selected here are not independent of model representation: action-chunk length changes intervention latency, multimodal action distributions represent multiple modes in offline data, and a generalist policy's action convention redefines both support and control rate. The next chapter therefore compares ACT, Diffusion Policy, RT-X, OpenVLA, and pi0 through action representation, latency, sensor channels, and recovery authority, connecting them back to the replay sets and release gates defined here.

References

1. Mandlekar, Ajay (2021). What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. arXiv.
2. Kalashnikov, Dmitry (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv.
3. Levine, Sergey (2016). Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. arXiv.
4. Dasari, Sudeep (2019). RoboNet: Large-Scale Multi-Robot Learning. arXiv.
5. Ebert, Frederik (2021). Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets. arXiv.
6. Brohan, Anthony (2022). RT-1: Robotics Transformer for Real-World Control at Scale. arXiv.
7. Khazatsky, Alexander (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. arXiv.
8. O'Neill, Abby (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv.
9. Zhao, Tony Z. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. arXiv.
10. Chi, Cheng (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv. #34
11. Chi, Cheng et al. (2024). Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. arXiv.
12. Choi, Hojung (2026). In-the-Wild Compliant Manipulation with UMI-FT. arXiv.
13. Xu, Mengda (2025). DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation. arXiv.
14. Fang, Hao-Shu (2025). DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation. arXiv.
15. Si, Zilin (2025). ExoStart: Efficient learning for dexterous manipulation with sensorized exoskeleton demonstrations. arXiv. #9
16. Li, Xingyu (2024). Evaluating Real-World Robot Manipulation Policies in Simulation. arXiv.
17. NVIDIA (2025). Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. NVIDIA Research.
18. Toyota Research Institute (2024). Large Behavior Models for Robot Manipulation. Company technical post.
19. Black, Kevin (2024). pi0: A Vision-Language-Action Flow Model for General Robot Control. arXiv.
20. Octo Model Team (2024). Octo: An Open-Source Generalist Robot Policy. arXiv.
21. Physical Intelligence (2025). OpenPI: Open Source Robot Policy Stack. GitHub.
22. AgiBot-World Contributors et al. (2025). AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems. arXiv.
23. Yu, Wenhao (2025). ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation. arXiv.
24. Feng, Ruoxuan (2025). AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-Tactile Sensors. arXiv.
25. Ross, Stéphane et al. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS.
26. Levine, Sergey et al. (2015). Learning Contact-Rich Manipulation Skills with Guided Policy Search. ICRA.
27. Florence, Pete et al. (2021). Implicit Behavioral Cloning. CoRL.
28. Kumar, Aviral et al. (2020). Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS.
29. Nair, Ashvin et al. (2020). AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. arXiv.
30. Yu, Tianhe et al. (2020). MOPO: Model-based Offline Policy Optimization. NeurIPS.
31. Gulcehre, Caglar et al. (2020). RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning. NeurIPS.
32. Hoque, Ryan et al. (2021). ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning. CoRL.
33. Liu, Huihan et al. (2023). Model-Based Runtime Monitoring with Interactive Imitation Learning. arXiv.
34. Luo, Jianlan et al. (2025). Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Science Robotics.
35. Luo, Jianlan et al. (2024). SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning. arXiv.
36. Yu, Tianhe et al. (2021). MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. arXiv.
37. Lu, Yao et al. (2022). AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale. PMLR.
38. Wu, Philipp et al. (2023). DayDreamer: World Models for Physical Robot Learning. CoRL.
39. Achiam, Joshua et al. (2017). Constrained Policy Optimization. ICML.
40. Rudin, Nikita et al. (2021). Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. CoRL.
41. Siekmann, Jonah et al. (2021). Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning. RSS.
42. Jiang, Zhenyu et al. (2024). DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning. arXiv.
43. Chebotar, Yevgen et al. (2019). Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience. ICRA.
44. Chen, Claire et al. (2025). DexForce: Extracting Force-Informed Actions from Kinesthetic Demonstrations. IEEE Robotics and Automation Letters. #3
45. Physical Intelligence (2025). π*₀.₆: a VLA That Learns From Experience. arXiv. #4 · KO
46. Terry Um (2026). ENPIRE: Robot Policy Self-Improvement. Terry's Blog. #69 · KO
47. Makoviychuk, Viktor et al. (2021). Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. arXiv.
48. Sergey Levine et al. (2016). End-to-End Training of Deep Visuomotor Policies. Journal of Machine Learning Research.
49. Chelsea Finn et al. (2017). One-Shot Visual Imitation Learning via Meta-Learning. CoRL.
50. John Schulman et al. (2017). Proximal Policy Optimization Algorithms. arXiv preprint.
51. Andy Zeng et al. (2021). Transporter Networks: Rearranging the Visual World for Robotic Manipulation. CoRL.
52. Eric Jang et al. (2022). BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning. CoRL.
53. Niklas Funk et al. (2025). On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lighting. arXiv preprint.
54. Anusha Nagabandi et al. (2018). Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning. IEEE ICRA.