Summary
MILER is an end-to-end reinforcement learning policy framework for autonomous driving that achieves zero-shot sim-to-real transfer, presented in arXiv paper 2609.20747 by Thomas Steinecker, Denis Trescher, Alexander Bienemann, Thorsten Luettel, and Mirko Maehlisch. The approach trains a policy network offline in a custom semantic mid-level representation (MLR) simulator, with control outputs applied to a bicycle model. During deployment, camera and LiDAR data are processed by BEVFusion to produce a semantic bird's-eye-view representation consistent with the simulator, and a trajectory alignment strategy bridges perception and control without direct actuation of raw policy outputs. The system was evaluated on test roads featuring diverse obstacles, hairpin turns, speeds up to 33.6 km/h, and off-road sections. Two different vehicles autonomously drove 17.3 km combined over a 3.0 km test course without human intervention. The full software stack runs on a Jetson AGX Orin. This work addresses a key barrier to applying reinforcement learning in unstructured real-world environments: sim-to-real transfer.
Paper Overview
- Field: Machine Learning
- Authors: Thomas Steinecker, Denis Trescher, Alexander Bienemann, Thorsten Luettel, Mirko Maehlisch
- Published: 2026-09-17
- arXiv: 2609.20747
Abstract
Reinforcement learning constitutes a promising approach owing to its potential for superhuman performance and self-learned policies. However, its application to real-world autonomous driving remains scarce, particularly in unstructured environments, because of the challenges associated with sim-to-real transfer.
This paper presents MILER, an end-to-end policy framework with zero-shot sim-to-real transfer:
- Training: A custom semantic mid-level representation (MLR) simulator is used offline; the policy network is trained with reinforcement learning, and its control outputs are applied directly to a bicycle model.
- Deployment: Camera and LiDAR data are processed by BEVFusion to generate a semantic bird's-eye-view representation consistent with the MLR simulator. Policy actions are not applied directly to the real vehicle; instead, a trajectory alignment strategy enables zero-shot transfer of both perception and control.
- Evaluation: Testing covered diverse challenges including multiple obstacles, hairpin turns, speeds up to 33.6 km/h, and off-road sections. Two different vehicles drove a total of 17.3 km over a 3.0 km test course without any human intervention.
- Hardware: The entire software stack runs on a Jetson AGX Orin.
The results demonstrate that semantic mid-level representations combined with trajectory alignment can make reinforcement learning practical for real-world autonomous driving in unstructured environments.
---
*Auto-collected on 2026-09-20.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635008