Pictura: Scaling Perspective-View Self-Play Training for Autonomous Driving

Forum topic · 小凯 · 2026-07-30

Summary

Pictura is a GPU-accelerated multi-agent driving simulator introduced in an arXiv paper (2607.26005) that establishes perspective-view self-play as a practical training regime for driving policies. Existing large-scale self-play approaches rely on privileged vectorized observations such as exact poses and velocities, including for occluded agents, creating a representation gap with the egocentric camera views of deployed vehicles; distillation into camera-based students forces them to imitate decisions their own view cannot justify. Pictura addresses this at the source by rendering each agent's egocentric view at every simulation step, sustaining up to 500K agent steps per second (2M images per second) on a single H100 GPU. Using Pictura, the authors train Alberti with vanilla PPO self-play—the first large-scale driving self-play policy trained directly from perspective images without privileged observations. Training spans 50 billion agent steps, roughly 35 million kilometers of driving. Alberti approaches the driving performance of its privileged vectorized counterpart and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized agents.

Overview

Research area: Computer Vision (CV) Authors: Yuan Yin, Elias Ramzi, Marc Lafon, Valentin Charraut, Victor Bares, Yihong Xu, Éloi Zablocki, Alexandre Boulch, Thibault Buhet, Andrei Bursuc, Matthieu Cord Published: 2026-07-28 arXiv: 2607.26005

Abstract

Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed agent driving from the perspective view of egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime.

Key points

Pictura simulator: A GPU-accelerated multi-agent driving simulator that renders each agent's egocentric view at every step, mitigating the representation gap at its source.
Throughput: Sustains up to 500K agent steps per second (2M images per second) on a single H100 GPU.
Alberti policy: Trained with vanilla PPO self-play; the first large-scale driving self-play policy trained directly from perspective images, without privileged observations.
Scale: Training spans 50B agent steps, approximately 35 million kilometers of driving.
Results: Alberti approaches the driving performance of its privileged vectorized counterpart and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, outperforming privileged vectorized agents.

*Auto-collected on 2026-07-30.*