Robot learning

Robot learning studies how robots acquire or improve behavior from data and experience. It lies at the intersection of robotics and machine learning, but the physical setting changes the learning problem: a robot acts through imperfect hardware, receives partial and noisy observations, and must collect data under limits imposed by time, wear, safety, and human supervision. Robot learning therefore includes methods based on demonstrations, rewards, self-collected experience, simulation, and combinations of learned and model-based components.^[1][2][3]

The term describes a research area rather than one algorithm. A learned component may map sensor observations directly to actions, estimate a reward or value function, learn a reusable representation, or select among previously learned skills. Some systems are trained for one robot and task; others are pretrained on data from several tasks or robot embodiments and then adapted. Claims of generality remain evaluation-dependent because robots, action spaces, environments, and success criteria differ across studies.^[2][3]

Scope and formulation

A robot control policy maps information available to the robot, such as camera images, joint positions, forces, or a task instruction, to an action or distribution over actions. Depending on the system, an action can be a motor command, a desired joint position, an end-effector displacement, or a higher-level skill. Learning can occur online while the robot interacts with its environment, offline from a previously collected dataset, in simulation, or through a mixture of these settings.^[1][2]

Robot learning overlaps with several neighboring fields:

• Imitation learning trains behavior from expert examples. The broader learning-from-demonstration literature also studies how demonstrations are obtained, represented, and reused; terminology is not fully uniform across publications.^[1]
• Reinforcement learning optimizes behavior using reward feedback. In robotics, exploration and data collection must account for physical cost, delays, safety constraints, and limited resets.^[2]
• Computer vision and state estimation provide representations of the environment, while planning and control determine how desired behavior is executed.
• Representation learning, language modeling, and generative modeling can be components of a robot policy, but a model is not a robot policy merely because it processes images or text.

This overlap does not eliminate the role of mechanics or control. Learned perception and action modules are commonly combined with kinematic models, trajectory generators, feedback controllers, safety checks, or manually specified skill libraries.^[2][3]

Learning from demonstrations

In learning from demonstration, a teacher supplies examples of the behavior to be learned. Demonstrations may be recorded through teleoperation, kinesthetic teaching, motion capture, or observation. A dataset typically relates states or observations to the teacher's actions, but the details depend on what the robot can sense and how its action space is represented.^[1]

Behavioral cloning

Behavioral cloning treats recorded observation-action pairs as a supervised-learning dataset. A policy is trained to predict the demonstrator's action from the current observation, or from a history of observations. This direct formulation can avoid manual reward design, but it creates a distribution-shift problem. During deployment, a small prediction error may bring the robot to states that were rare or absent in the demonstrations, where further errors can compound.^[1][5]

ALVINN was an early example of a neural network mapping road images to steering directions. The 1988 paper trained the network with simulated road images and reported tests on images of real roads under some conditions. It should not be described as having been trained solely by watching a human driver; that characterization belongs to later work rather than the experiment documented in the 1988 paper.^[4]

DAgger, introduced by Ross, Gordon, and Bagnell in 2011, addresses distribution shift by repeatedly running the current policy, asking an expert to label the states the policy visits, and aggregating those labels with earlier training data. Its theoretical result is conditional: the paper reduces the sequential prediction problem to online learning and analyzes the method when the underlying online learner has low regret. It is not an unconditional guarantee that an arbitrary learned policy will match an expert.^[5]

Inverse reinforcement learning

Inverse reinforcement learning infers an objective from demonstrated behavior instead of directly predicting the demonstrator's next action. In the formulation of Abbeel and Ng, the reward is linear in known features. Their apprenticeship-learning method seeks a policy whose expected feature counts are close to the expert's, which can yield comparable performance for rewards in that class without uniquely recovering the expert's true reward.^[6]

This distinction matters in practice. Behavioral cloning asks how to reproduce demonstrated actions, while inverse reinforcement learning asks which objective could make the behavior desirable. Both depend on assumptions about observations, actions, demonstrations, and model class, and neither guarantees that an inferred objective captures every preference of a human demonstrator.^[1][6]

Reinforcement learning on robots

In reinforcement learning, a policy is optimized to increase expected cumulative reward through interaction. Robot applications often have continuous state and action spaces, delayed consequences, and expensive samples. Physical trials can damage hardware or nearby objects, and an experiment may require a person to reset the scene. Sensor drift, changing lighting, wear, communication delays, and differences between nominally identical robots also make the data nonstationary.^[2][8]

These conditions motivate several practical choices. Researchers may learn in simulation before transferring a policy, initialize a policy from demonstrations, reuse an offline dataset, constrain exploration with a controller, or fine-tune a pretrained policy. The resulting system can combine supervised learning, reinforcement learning, and conventional feedback control rather than fitting one exclusive category.^[2][3]

An influential real-robot example by Levine and colleagues learned deep visuomotor policies that mapped camera observations to motor torques. The work used guided policy search to connect trajectory optimization with policy training and evaluated the method on manipulation tasks using real robotic systems. It was a task-bounded demonstration of end-to-end visuomotor learning, not evidence that a single policy had solved manipulation generally.^[7]

Simulation and transfer to physical robots

Simulation allows repeated trials without placing the same burden on physical hardware. A simulator can also expose a learner to conditions that are difficult to reproduce safely. Transfer is difficult, however, because simulated images, contact behavior, actuator response, latency, and sensor noise do not exactly match the physical system.^[2]

Domain randomization trains across a distribution of simulated conditions rather than one fixed simulator configuration. Tobin and colleagues randomized visual properties in simulation and evaluated transfer for object localization and grasping. Peng and colleagues randomized physical dynamics and reported transfer of a robot-arm pushing policy without real-world training for that policy. These studies support domain randomization for their specified tasks; they do not establish that randomization removes every simulation mismatch.^[9][10]

A 2019 Rubik's Cube system combined reinforcement learning in simulation with automatic domain randomization and transferred the policy to a Shadow Dexterous Hand. The hand had 20 actuated joints. In the paper's physical evaluation, the system solved scrambles requiring 15 face rotations in 60 percent of 10 trials and scrambles requiring 26 face rotations in 20 percent of 10 trials. The authors noted that 26 quarter-face rotations are sufficient for any cube position. These are protocol-specific success rates, not a general accuracy figure for dexterous manipulation.^[11]

Transfer methods can also use system identification, adaptation from real-world data, residual corrections, or fine-tuning on the physical platform. Their suitability depends on what differs between simulation and reality and on how much safe real-world data is available.^[2][8]

Data collection and dataset design

Data is a central constraint because demonstrations from a physical robot are slower and more expensive to collect than many web or simulated datasets. Collection choices also shape what a policy can learn. Camera placement, robot embodiment, controller interface, operator behavior, scene layout, and task definitions can all introduce systematic differences.^[1][3]

DROID illustrates a distributed collection effort. Its authors reported 76,000 demonstration trajectories, or 350 hours of interaction, across 564 scenes and 84 tasks. The data was collected by 50 people in North America, Asia, and Europe over 12 months. Those figures describe the released dataset and collection protocol; they do not by themselves show that a policy will transfer to every scene or robot.^[12]

Dataset diversity and consistency create a tradeoff. More environments and operators can broaden coverage, while inconsistent action spaces, observation formats, task labels, and hardware may make joint training harder. Cross-embodiment projects therefore standardize data formats or learn model components that can accommodate different inputs and outputs.^[3][15]

Generalist and vision-language-action policies

Since 2022, several projects have trained robot policies on larger and more varied collections of demonstrations. Many accept images and a language instruction and produce robot actions. Publications often call this family vision-language-action, or VLA, models. The label covers different architectures and training mixtures, so model size or dataset size alone does not establish broader capability.

Scroll sideways for more →

RT-1 processes a short image history and a natural-language instruction to predict tokenized robot actions. Its visual encoder was initialized from ImageNet pretraining, so it is inaccurate to say that every part of the system was learned only from robot data.^[13]

RT-2 co-fine-tuned vision-language models on web data and robot trajectories, representing robot actions as tokens that could be handled by the model's text interface. The paper used the term vision-language-action model for this formulation and evaluated variants in more than 6,000 trials. Reported reasoning or generalization examples should be interpreted within those controlled evaluations rather than as unrestricted physical reasoning.^[14]

The Open X-Embodiment paper aggregated datasets from 22 robot embodiments through a collaboration of 21 institutions and reported 527 skills and 160,266 tasks. RT-X experiments showed positive transfer in the tested settings. The counts and results are specific to the paper's definitions and evaluations; they do not mean that all contributing datasets used identical task labels or action spaces.^[15]

Octo was trained on 800,000 trajectories and evaluated across nine robot platforms. It supports language goals or goal images and can be fine-tuned to new observation and action spaces. OpenVLA combined a Llama 2 language model with visual features from DINOv2 and SigLIP. Its authors reported comparisons across 29 tasks and multiple embodiments, including a 16.5 percentage-point absolute advantage over RT-2-X in that evaluation. These are results from different studies, so their headline numbers should not be compared without accounting for tasks, robots, training data, and evaluation protocol.^[16][17]

The pi0 paper paired a pretrained vision-language model with an action expert trained using flow matching. It reported evaluations on multiple robot platforms and tasks including laundry folding, table cleaning, and assembling boxes. The paper supports those bounded experiments, but not a claim that one released policy can control arbitrary hardware or complete arbitrary instructions.^[18]

Language-conditioned robot systems

Language can enter a robot system at different levels. In SayCan, a language model proposed useful high-level skills while learned value functions scored how feasible those skills were in the current physical situation. The combination grounded language-based selection in skills available to a mobile manipulator. SayCan therefore selected among learned skills rather than directly generating every low-level motor command.^[19]

PaLM-E took a different approach. It interleaved encodings of continuous sensor inputs, including images and state estimates, with text inside a language-model architecture and trained the system on a mixture of embodied and general visual-language tasks. It is more accurate to say that sensor observations were mapped into the model's embedding space than to call them "hardware embeddings."^[20]

VLA policies such as RT-2, OpenVLA, and pi0 use language and visual observations in systems that produce robot actions, but their action representations differ. These systems should not be treated as interchangeable with high-level planners such as SayCan or with multimodal reasoning models that do not themselves define a low-level controller.^{[14][17][18][19][20]}

Manipulation and locomotion examples

Robot manipulation includes reaching, grasping, moving articulated objects, tool use, and bimanual tasks. These problems involve contact, occlusion, and multiple valid action sequences. A learned policy may need to represent a multimodal action distribution rather than average incompatible strategies.^[3]

Diffusion Policy represents a visuomotor policy as a conditional denoising diffusion process over actions. Its authors evaluated it on 12 tasks from four robot-manipulation benchmarks and reported an average improvement of 46.9 percent over the baselines used in that study. That number is an aggregate within the paper's benchmark suite, not a universal improvement over other policies.^[21]

ALOHA is a low-cost bimanual teleoperation system paired in its initial paper with an Action Chunking with Transformers policy. The authors evaluated six real-world tasks and reported success rates of 80 to 90 percent using demonstrations collected for each task. Those experiments show the utility of the specific hardware, data collection, and policy combination; they do not establish the same performance for other tasks or robots.^[22]

Locomotion research similarly combines learned policies with physical design and control structure. Radosavovic and colleagues trained a causal transformer policy for a Digit humanoid using large-scale, model-free reinforcement learning in randomized simulation. They reported zero-shot transfer to the physical robot and evaluated walking over outdoor terrain and responses to disturbances. The result concerns that platform, training procedure, and evaluation, rather than humanoid locomotion as a whole.^[23]

Evaluation and limitations

Robot-learning results are difficult to compare across papers because evaluation protocols vary. A success rate depends on the initial-state distribution, number of trials, task tolerance, human interventions, scene setup, and whether tasks or objects appeared in training. Aggregate results may also hide large differences among individual tasks.^[3][8]

Important evaluation questions include:

• Was the policy tested on the same robot embodiment used for training?
• Were objects, scenes, and instructions held out, and how was that split constructed?
• How many physical trials were run, and were failures or human resets counted consistently?
• Does the policy run at the control rate required by the hardware?
• What safety controller, planner, or recovery procedure operated alongside the learned model?
• Can the experiment be reproduced from released data, code, hardware details, and evaluation instructions?

Real-world deployment adds problems that a benchmark may not capture, including changing conditions, delayed rewards, partial observability, safety constraints, hardware maintenance, and the need for continual monitoring.^[8] A system that succeeds in a laboratory evaluation can still require substantial engineering before reliable use outside that setup.

Relationship to classical robotics

Robot learning and classical robotics are not a strict either-or choice. Classical methods use models of kinematics and dynamics, planning algorithms, state estimators, and feedback control. Learning methods estimate behavior or representations from data. A practical system can use a learned perception model with a conventional planner, a learned high-level selector with verified low-level controllers, or a simulation-trained policy inside a safety envelope.^[2][3][19]

The useful distinction is therefore not whether a robot contains equations or a neural network, but which parts are learned, which assumptions are encoded by engineers, what data supports the learned parts, and how the complete system is evaluated on physical hardware.

See also

• Robotics
• Machine learning
• Reinforcement learning
• Imitation learning
• Computer vision
• Foundation models
• Transformer
• Diffusion models
• Open X-Embodiment

References

1. ^Argall, B. D., Chernova, S., Veloso, M., and Browning, B. (2009). A survey of robot learning from demonstration. *Robotics and Autonomous Systems*, 57(5), 469-483. eecs.northwestern.edu/...09ras_argall.pdf
2. ^Kober, J., Bagnell, J. A., and Peters, J. (2013). Reinforcement learning in robotics: A survey. *The International Journal of Robotics Research*, 32(11), 1238-1274. ri.cmu.edu/...Kober_IJRR_2013.pdf
3. ^Kroemer, O., Niekum, S., and Konidaris, G. (2021). A review of robot learning for manipulation: Challenges, representations, and algorithms. *Journal of Machine Learning Research*, 22(30), 1-82. jmlr.org/...19-804
4. ^Pomerleau, D. A. (1988). ALVINN: An autonomous land vehicle in a neural network. *Advances in Neural Information Processing Systems 1*. proceedings.neurips.cc/...9d43bbf5bbe87fb-Abstract
5. ^Ross, S., Gordon, G. J., and Bagnell, J. A. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. *Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics*, 627-635. proceedings.mlr.press/...ross11a
6. ^Abbeel, P., and Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. *Proceedings of the Twenty-first International Conference on Machine Learning*. ai.stanford.edu/...AbbeelNg_alvirl_ICML2004.pdf
7. ^Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2016). End-to-end training of deep visuomotor policies. *Journal of Machine Learning Research*, 17(39), 1-40. jmlr.org/...15-522
8. ^Dulac-Arnold, G., et al. (2019). Challenges of real-world reinforcement learning. arXiv:1904.12901. arxiv.org/...1904.12901
9. ^Tobin, J., et al. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. arXiv:1703.06907. arxiv.org/...1703.06907
10. ^Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. *2018 IEEE International Conference on Robotics and Automation*, 3803-3810. arxiv.org/...1710.06537
11. ^Akkaya, I., et al. (2019). Solving Rubik's Cube with a robot hand. arXiv:1910.07113. arxiv.org/...1910.07113
12. ^Khazatsky, A., et al. (2024). DROID: A large-scale in-the-wild robot manipulation dataset. arXiv:2403.12945. arxiv.org/...2403.12945
13. ^Brohan, A., et al. (2022). RT-1: Robotics Transformer for real-world control at scale. arXiv:2212.06817. arxiv.org/...2212.06817
14. ^Zitkovich, B., et al. (2023). RT-2: Vision-language-action models transfer web knowledge to robotic control. *Proceedings of The 7th Conference on Robot Learning*, 2165-2183. proceedings.mlr.press/...zitkovich23a
15. ^Open X-Embodiment Collaboration. (2023). Open X-Embodiment: Robotic learning datasets and RT-X models. arXiv:2310.08864. arxiv.org/...2310.08864
16. ^Octo Model Team, et al. (2024). Octo: An open-source generalist robot policy. *Robotics: Science and Systems XX*. roboticsproceedings.org/...p090
17. ^Kim, M. J., et al. (2024). OpenVLA: An open-source vision-language-action model. arXiv:2406.09246. arxiv.org/...2406.09246
18. ^Black, K., et al. (2024). pi0: A vision-language-action flow model for general robot control. arXiv:2410.24164. arxiv.org/...2410.24164
19. ^Ahn, M., et al. (2023). Do As I Can, Not As I Say: Grounding language in robotic affordances. *Proceedings of The 6th Conference on Robot Learning*, 287-318. proceedings.mlr.press/...ichter23a
20. ^Driess, D., et al. (2023). PaLM-E: An embodied multimodal language model. *Proceedings of the 40th International Conference on Machine Learning*, 8469-8488. proceedings.mlr.press/...driess23a
21. ^Chi, C., et al. (2023). Diffusion Policy: Visuomotor policy learning via action diffusion. *Robotics: Science and Systems XIX*. roboticsproceedings.org/...p026
22. ^Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. (2023). Learning fine-grained bimanual manipulation with low-cost hardware. arXiv:2304.13705. arxiv.org/...2304.13705
23. ^Radosavovic, I., et al. (2024). Real-world humanoid locomotion with reinforcement learning. *Science Robotics*, 9(89), eadi9579. pubmed.ncbi.nlm.nih.gov/38630806