Learn · Beginner

Imitation Learning: Teaching Machines by Showing, Not Rewarding

Imitation learning trains a system by copying demonstrations instead of discovering behaviour through trial and error. Its simplest form, behaviour cloning, is nothing more than supervised learning applied to actions: collect examples of an expert's situation-and-response pairs, then train a network to predict the response from the situation. It is how most modern robots are taught, how chat models learn to answer questions, and it comes with one deep, well-understood flaw.

The appeal is that it sidesteps the hardest part of reinforcement learning: the reward function. To train a robot to fold a shirt by reward alone, you must write a mathematical expression scoring how folded a shirt is - and every attempt at that produces something a policy can game rather than satisfy. Demonstrations dodge the question. Nobody has to define good folding if a person can simply fold twenty shirts while the sensors record.

The technique goes back further than most people expect. Dean Pomerleau's ALVINN drove a van on public roads in 1989 using a small neural network trained on camera images paired with the steering angles a human driver chose. NVIDIA revived the approach in 2016 with an end-to-end self-driving system trained on 72 hours of human driving footage, mapping raw camera pixels straight to steering commands with no hand-written lane detection anywhere in the pipeline.

And then the flaw. A behaviour-cloned policy only ever saw states the expert visited, and experts stay on the good path. The moment the policy makes a small error it lands slightly off that path, in a state its training data never contained, where its output is essentially a guess. That guess pushes it further off. The next state is stranger still. Stephane Ross, Geoffrey Gordon and Drew Bagnell formalised this in 2010 and showed the damage grows quadratically with the length of the episode rather than adding up linearly - a policy accurate 99% of the time per step does not stay 99% accurate over a thousand steps.

The analogy is learning to ride a bicycle by watching video of people riding bicycles. You will absorb what balanced riding looks like and nothing about what to do when you start to tip, because the videos never show anyone tipping. The first wobble puts you somewhere your training never went.

The fixes all amount to getting the expert to demonstrate recovery. DAgger, from the same 2010 paper, runs the learned policy, collects the states it actually reaches, and asks the expert to label the right action in each - then retrains, and repeats. NVIDIA's driving system used a cruder version: two extra cameras pointed slightly left and right, with steering labels adjusted to correct back toward the centre, manufacturing recovery data without ever crashing a car. Jonathan Ho and Stefano Ermon took a different route in 2016 with generative adversarial imitation learning, training a discriminator to distinguish expert behaviour from policy behaviour and using its judgement as a learned reward - the same adversarial idea behind GANs, pointed at actions.

The modern robotics revival changed what gets predicted rather than how data is collected. Cheng Chi and colleagues introduced Diffusion Policy in 2023, generating a short sequence of future actions with a diffusion model rather than a single action with a regression head. That matters more than it sounds: when a demonstration set contains two valid ways to do something - go around the mug on the left or the right - a network trained to predict the average action outputs the average of left and right, which drives straight into the mug. A diffusion model represents both modes and samples one. Tony Zhao and colleagues made a related move the same year with action chunking, predicting a block of future actions at once so the policy commits to a coherent plan instead of dithering step to step. Both ideas are now standard in vision-language-action models.

Language models are the largest deployment of behaviour cloning in existence, though it is rarely called that. Supervised fine-tuning on human-written responses is exactly behaviour cloning with text: the expert is the human writer, the state is the conversation so far, the action is the next token. It inherits both properties. It is fast, stable and remarkably effective - and it never teaches the model what to do after it has said something wrong, which is one reason reinforcement learning from human feedback and verifiable-reward training were layered on top.

The honest caveats are three. A cloned policy is capped by its demonstrator; copying mediocre driving produces mediocre driving. Demonstration data is expensive, because a person must physically perform every example, which is why so much robotics work goes into teleoperation rigs and sim-to-real transfer. And a policy that scores well on held-out demonstrations can still fail in deployment, because held-out demonstrations are drawn from expert states while deployment happens in the states the policy itself creates. That last gap is the entire subject, and it has not been closed - only managed. Related reading: Markov decision processes, on-policy vs off-policy learning, and reward hacking.

Ground Truth. (2026, August 1). Imitation Learning: Teaching Machines by Showing, Not Rewarding. Ground Truth. https://groundtruth.day/learn/imitation-learning-and-behavior-cloning.html

@misc{groundtruth:imitation-learning-and-behavior-cloning,
  title  = {Imitation Learning: Teaching Machines by Showing, Not Rewarding},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/imitation-learning-and-behavior-cloning.html}
}

Topics: reinforcement-learning · robotics · training · agents · vla