OPTED: On-Policy Fine-Tuning for End-to-End Autonomous Driving with a Teacher-Student RL Framework

Forum topic · 小凯 · 2026-09-20

Summary

OPTED (On-Policy Fine-Tuning for End-to-End Driving) is a method that decouples reinforcement learning from the post-training of end-to-end driving policies. Since behavior-cloning pre-training suffers from compounding errors in closed-loop deployment, closed-loop post-training is desirable, but sensor-based policies normally require costly simulation for RL. OPTED instead trains a privileged teacher with RL on vectorized inputs (HD map and bounding boxes), then uses this teacher to supervise a pre-trained camera-based student policy during closed-loop post-training. Applied to TransFuser and VaVAM with fine-tuning in AlpaSim using neural reconstructions (3D Gaussian Splatting) of real driving logs, OPTED improves driving scores by 1.6x and 9.5x respectively. Controlled experiments show it matches the closed-loop performance of direct RL post-training with roughly three orders of magnitude fewer simulator interactions, while staying closer to human priors. Source: arXiv:2609.20756.

Paper Overview

Field: Computer Vision Authors: Damiano Da Col, Maximilian Igl, Peter Karkus, Kashyap Chitta, Boris Ivanovic, Marco Pavone, Konrad Schindler, Christos Sakaridis Published: 2026-09-17 arXiv: 2609.20756

Abstract

As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies.

OPTED (on-policy fine-tuning for end-to-end driving) decouples reinforcement learning from the post-training of the end-to-end policy:

1. A privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). 2. The teacher then provides supervision for a pre-trained student policy during closed-loop post-training.

Results

The authors apply OPTED to two camera-based end-to-end models, TransFuser and VaVAM, fine-tuned in AlpaSim using neural reconstructions (3D Gaussian Splatting) of real driving logs:

TransFuser: driving score improved by 1.6x
VaVAM: driving score improved by 9.5x

Controlled experiments show that OPTED achieves the same closed-loop performance as direct RL post-training with roughly three orders of magnitude fewer simulator interactions, while producing policies that remain closer to human priors.

--- *Auto-collected on 2026-09-20.*