Skip to content
mlmentorship

Robotics policy learning

Learn actions from demonstrations, rewards, or world models while respecting partial observability, control frequency, safety, and sim-to-real shift.

Published · 6 min read ·Specialist ·Advanced

Visual quick review

Visual first · depth when needed

Explain why behavior cloning's training distribution can stop matching the states its learned policy visits, and trace how DAgger uses expert labels on those visited states to reduce the mismatch.

Preparing the visual…

Summary

Robotics policy learning maps observations and goals to actions or action distributions using demonstrations, reinforcement learning, planning, or combinations of learned world models and control.

Robotics turns prediction error into physical consequence. Data is expensive, observations are partial, actions are continuous and temporally coupled, and small perception or control errors compound over a trajectory.

A visually impressive rollout says little without task distribution, intervention rate, safety constraints, and repeated trials.

Problem formulation

Define:

  • observation: cameras, proprioception, force, audio, language, history;
  • state belief: what the policy infers but cannot directly observe;
  • goal: language instruction, target state, reward, or reference trajectory;
  • action: joint targets, torques, end-effector commands, or action chunks;
  • control frequency and latency;
  • horizon and termination;
  • constraints and safety envelope;
  • environment distribution.

Action representation is a first-order choice. High-level waypoints simplify learning but rely on a low-level controller. Raw torques offer control but greatly expand difficulty and risk.

Learning approaches

Behavior cloning

Supervise the policy on expert observation-action pairs. It is simple and stable but suffers from covariate shift: small errors move the robot into states absent from demonstrations.

DAgger-style data collection asks an expert to label states visited by the learned policy, reducing this mismatch at the cost of interactive supervision.

Learning objective: explain why a behavior-cloned policy can leave its demonstration distribution, then trace how DAgger turns learner-visited states into new supervised examples.

Learning objective

Why can one imitation error compound, and what does DAgger change?

Behavior cloning can leave demonstrated states, while DAgger adds labels from states the learner visits In the upper panel, a solid expert path passes through four demonstrated states. A dashed learned-policy path initially follows it, then a small action error sends the robot to two states absent from the demonstrations, where later errors can compound. In the lower panel, DAgger repeatedly runs the current policy, asks the expert to label the visited states, aggregates those state-action pairs with earlier data, and retrains the policy. 1 · BEHAVIOR CLONING: POLICY-INDUCED SHIFT EXPERT DEMONSTRATIONS s₁ s₂ s₃ s₄ LEARNED POLICY s₁ s₂ s′₃ s′₄ small action error changes the next state NO DEMO LABELS 2 · DAGGER: LABEL LEARNER-VISITED STATES 1 · VISIT run current policy 2 · LABEL expert action at s′ 3 · AGGREGATE add (s′, expert action) 4 · RETRAIN fit next policy
Read it this way: follow the solid expert path first. Behavior cloning learns labels on those states, but its own action at s₂ determines the next state; a small error can land at s′₃, where the original demonstrations provide no corrective label, so later errors can compound. DAgger closes that gap iteratively: run the evolving policy, ask the expert what to do at the states it actually visits, add those pairs to the dataset, and retrain. The sketch is qualitative, not a claim that every deviation is unrecoverable. Original synthesis informed by Ross and Bagnell (2010) and Ross, Gordon, and Bagnell (2011).

Offline RL

Learn from fixed logged trajectories while estimating value beyond imitation. Distributional shift and extrapolation to unsupported actions are central risks.

Online RL

Interact to optimize reward. It can discover behavior beyond demonstrations but carries sample, safety, and reward-design costs.

Diffusion and sequence policies

Model a distribution over action sequences or chunks. Multimodal action distributions can represent several valid ways to complete a task. Chunking reduces decision frequency but can make rapid correction harder.

World models and planning

Predict future observations or latent states and plan actions against the model. Planning quality is bounded by model error, especially off the data distribution.

Vision-language-action models

Use large vision-language representations to condition robot actions. They can transfer semantic knowledge, but embodiment, geometry, timing, and safety still require robot-specific data and evaluation.

Sim-to-real

Simulation offers cheap and safe data but differs in dynamics, sensing, contact, appearance, and latency. Techniques include domain randomization, system identification, representation adaptation, residual policies, and real-world fine-tuning.

Randomization helps only if the real system lies within the randomized support. Unrealistic diversity can also make learning harder.

Evaluation

Report:

  • success and partial progress by task and environment slice;
  • intervention and safety-violation rates;
  • time, path length, energy, and object damage;
  • robustness to lighting, viewpoint, object, and dynamics shift;
  • recovery after perturbation;
  • calibration or abstention when the policy is uncertain;
  • repeated trials and confidence intervals;
  • real versus simulated performance.

Separate perception, planning, and control failures when possible.

Common confusions

  • “Behavior cloning is enough with more data.” Policy-induced state shift can remain.
  • “A simulator removes safety concerns.” Deployment still faces unmodeled dynamics and hardware limits.
  • “Language understanding solves control.” Semantic goals do not supply precise geometry or stable feedback control.
  • “Success rate captures safety.” A successful trajectory can contain near collisions or excessive force.
  • “End-to-end means no structure.” Action interfaces, safety controllers, and planners still impose structure.
  • “One demo video proves generalization.” Repeat across controlled shifts and seeds.

In an interview

Define observation, action, frequency, horizon, safety, and data source. Then choose imitation, RL, planning, or a hybrid, and design evaluation that separates task success from intervention, robustness, and physical risk.

Related: multimodal foundation models, reward shaping, and domain adaptation.

Found an error or stale claim? Report a correction or send a private note.