OPTED: Render-Free On-Policy Fine-Tuning for End-to-End Autonomous Driving

Forum topic · 小凯 · 2026-09-20

Summary

OPTED (on-policy fine-tuning for end-to-end driving) is a post-training framework that decouples reinforcement learning from the fine-tuning of sensor-based end-to-end driving policies. Instead of running costly RL in sensor simulation, OPTED first trains a privileged teacher with RL on vectorized inputs such as HD maps and bounding boxes, then uses this teacher to supervise a pre-trained vision-based student during closed-loop post-training. The authors apply OPTED to two camera-based models, TransFuser and VaVAM, fine-tuning them in AlpaSim using neural reconstructions (3D Gaussian Splatting) of real driving logs. Driving scores improve by 1.6x for TransFuser and 9.5x for VaVAM. Controlled experiments show that OPTED achieves the same closed-loop performance as direct RL post-training while requiring roughly three orders of magnitude fewer simulator interactions, and produces policies closer to human priors. Paper: arXiv 2609.20756.

Overview

As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies.

Method

OPTED (On-Policy fine-Tuning for End-to-end Driving) decouples reinforcement learning from the post-training of the end-to-end policy:

1. A privileged teacher is trained with RL on vectorized inputs (HD map and bounding boxes). 2. The teacher then supervises a pre-trained vision-based student during closed-loop on-policy fine-tuning, avoiding expensive sensor-level simulation for RL.

Results

The method was applied to two camera-based models, TransFuser and VaVAM, fine-tuned in AlpaSim with neural reconstructions (3DGS) of real driving logs:

TransFuser: driving score improved by 1.6x
VaVAM: driving score improved by 9.5x

Controlled experiments show that OPTED reaches the same closed-loop performance as direct RL post-training with roughly three orders of magnitude fewer simulator interactions, while producing policies that stay closer to human priors.