MILER: Semantic Mid-Level Representation for Zero-Shot Sim-to-Real Reinforcement Learning in Autonomous Driving

Forum topic · 小凯 · 2026-09-20

Summary

MILER is an end-to-end reinforcement learning framework for autonomous driving that achieves zero-shot sim-to-real transfer, proposed by Thomas Steinecker, Denis Trescher, Alexander Bienemann, Thorsten Luettel, and Mirko Maehlisch (arXiv:2609.20747). During offline training, a policy network is trained with reinforcement learning in a custom semantic mid-level representation (MLR) simulator, with control outputs applied to a bicycle model. At deployment, camera and LiDAR data are fused by BEVFusion into a semantic bird's-eye-view representation matching the MLR simulator. Instead of directly actuating the vehicle, the policy's actions are converted via a trajectory alignment strategy, enabling zero-shot transfer of both perception and control. The system was evaluated on test courses with various obstacles, hairpin turns, speeds up to 33.6 km/h, and off-road sections. Two different vehicles completed 17.3 km of autonomous driving without human intervention over a 3.0 km test track. The entire software stack runs on an NVIDIA Jetson AGX Orin. The work addresses the scarcity of real-world RL deployment in unstructured environments, where sim-to-real transfer is the main bottleneck.

Overview

Field: Machine Learning Authors: Thomas Steinecker, Denis Trescher, Alexander Bienemann, Thorsten Luettel, Mirko Maehlisch Published: 2026-09-17 arXiv: 2609.20747

Abstract

Reinforcement learning constitutes a promising approach owing to its potential for superhuman performance and self-learned policies. However, its application to real-world autonomous driving remains scarce, particularly in unstructured environments, because of the challenges associated with sim-to-real transfer for unstructured environments. In this work, the authors present MILER, an end-to-end policy framework with zero-shot sim-to-real transfer.

Method

Offline training: A custom semantic mid-level representation (MLR) simulator is used to train the policy network with reinforcement learning. Control outputs are applied directly to a bicycle model.
Real-vehicle deployment: Camera and LiDAR data are processed by BEVFusion to generate a semantic bird's-eye-view representation consistent with the MLR simulator.
Trajectory alignment: Actions produced by the policy network are not applied directly to the real vehicle; instead, a trajectory alignment strategy enables zero-shot transfer of both perception and control.

Evaluation

Testing covered diverse challenges: various obstacles, hairpin turns, speeds up to 33.6 km/h, and off-road sections.
Two different vehicles drove a total of 17.3 km without human intervention on a 3.0 km test track.
The full software stack runs on a Jetson AGX Orin.