Why Your Robot Drives Off Cliffs: A Guide to Immitation Learning

The Paradox

Here's what makes this maddening:

• Training accuracy: very high ✓
• Deployment success rate: often poor under covariate shift ✗

How does near-perfect training translate to catastrophic failure?

Let me show you what's happening. At timestep 1-30, the robot drives perfectly—smooth steering, following the centerline, exactly mimicking the expert. Then at timestep 31, it makes a tiny mistake. Just 0.3 meters off center.

No big deal, right?

Wrong.

By timestep 32, it's 0.8 meters off. Timestep 35: completely off track. Timestep 40: in freefall.

The issue? The robot has never seen states where it's off track. Every state in the training data shows the expert driving perfectly on the centerline. The moment the robot deviates even slightly, it enters completely uncharted territory.

And it has no idea what to do.

The Root Cause: Distribution Shift

This is called covariate shift, and it's unique to robotics.

In supervised learning, we assume the training distribution equals the test distribution. For images or text, this mostly holds. A photo of a cat is a photo of a cat, whether you saw it during training or not.

But robots are different. The robot's own actions change its future state distribution.

Training states (from expert): [On track, centered] → [On track, centered] → [On track, centered] Test states (from robot): [On track, centered] → [Slightly off] → [Way off] → [FALLING] ↑ NEW STATE

One small mistake at timestep t creates a distribution at timestep t+1 that wasn't in the training data. The robot makes another mistake. This creates an even worse distribution at t+2.

It's a death spiral.

The mathematical reality is harsh: this compounds quadratically.

For a 100-step episode with 1% per-step error:

• Expected cumulative error: 0.01 × 100² = 100

That's 10,000% error accumulation from what looked like a tiny training error.

This is why supervised learning—which works brilliantly for static tasks—fails catastrophically for robots.

The Fix: DAgger

What if instead of training on the expert's states, you trained on the learner's states?

This is the key insight behind DAgger (Dataset Aggregation).

The algorithm is beautifully simple:

1. Drive: Robot drives using its current policy.
2. Label: Human expert labels the states the robot actually visits.
3. Aggregate: Add these labeled states to the training dataset.
4. Retrain: Update the policy.
5. Repeat.

The magic is in step 3: aggregation. You don't discard the old data—you accumulate it. After a few iterations, your dataset contains:

• Expert's normal driving states
• Robot's mistake states + recovery actions
• Robot's recovery states + return-to-normal actions

Now your robot knows how to recover from mistakes.

Why This Works

DAgger eliminates distribution shift by ensuring your training data actually covers the states you'll encounter at test time. The error bound drops from O(εT²) to O(εT)—from quadratic to linear.

Real Results

Going Deeper: Not All Mistakes Are Equal

DAgger achieves linear error scaling, which is theoretically optimal. But there's still room for improvement in practice.

Consider two scenarios:

• Mistake A: Steering 0.5° left instead of 0.3° left while centered. (Easy to recover).
• Mistake B: Steering 0.5° left instead of 0.5° right at the cliff edge. (Game Over).

Standard DAgger treats both equally. This is where AggreVaTe (Aggregate Values to Imitate) comes in.

The Insight: Advantages

AggreVaTe minimizes advantage-weighted action error. The advantage function tells you: "If I take action 'a' instead of the expert's action, how much worse will my future be?"

Near the cliff, the advantage (penalty) is massive. The model learns: "Near the cliff, I need to get this exactly right."

The Unifying Framework

Every imitation learning algorithm is solving the same underlying game: min_π max_f [f(π) - f(π*)]. This is essentially GANs for policies.

When to Use What

• Use Behavior Cloning when: Almost never for robotics.
• Use DAgger when: You can query an expert in simulation or lab settings.
• Use AggreVaTe when: Some mistakes are catastrophic (Self-driving, Medical).
• Use Intervention Learning when: The expert is a human safety driver.
• Use Inverse RL when: You only have offline videos/data.

The Implementation

The single most critical line in DAgger:

# WRONG (catastrophic forgetting) self.dataset = new_data # RIGHT (aggregation) self.dataset.append(new_data)

That one method call—append() vs =—is the difference between failure and success. Aggregation ensures you keep improving on all states—normal and recovery.

Complete implementation and documentation can be found on my GitHub (RL_immitation_learing).

Key Takeaways

1. Distribution shift is the killer: Your actions change what states you see.
2. Interactive learning is the solution: DAgger queries experts on actual learner states.
3. Not all mistakes are equal: AggreVaTe weights errors by their consequences.
4. It's all one game: All these algorithms are min-max optimization variants.
5. The fix can be simple: Often it's just one line of code.

Further Reading

• (Ross & Bagnell (2011): Original DAgger paper)
• (Sanjiban Choudhury’s Lecture Series: 10 lectures from theory to practice.)