HERMES++: A Unified Driving World Model for 3D Scene Understanding and Future Geometry Prediction

Forum topic · 小凯 · 2026-05-02

Summary

HERMES++ is a unified driving world model that integrates 3D scene understanding and future geometric prediction within a single framework, addressing the gap between semantic interpretation (LLM-based reasoning) and physical simulation (future scene generation). The model introduces a co-designed architecture: BEV representations that consolidate multi-view spatial information into an LLM-compatible structure; LLM-enhanced world queries that enable knowledge transfer to the understanding branch; a Current-to-Future Link module bridging temporal gaps so geometry evolves conditioned on semantic context; and a joint geometric optimization strategy combining explicit geometric constraints with implicit latent-space regularization to align internal representations with geometry-aware priors. Experiments across multiple benchmarks show HERMES++ outperforms specialized methods on both future point cloud prediction and 3D scene understanding tasks. Paper: arXiv 2604.28196.

Paper Overview

Field: Computer Vision (CV) Authors: Yihang Qiu, Binglu Wang, Chenming Wu et al. arXiv: 2604.28196

Abstract

Driving world models serve as a pivotal technology for autonomous driving by simulating environmental dynamics. However, existing approaches predominantly focus on future scene generation, often overlooking comprehensive 3D scene understanding. Conversely, while Large Language Models (LLMs) demonstrate impressive reasoning capabilities, they lack the capacity to predict future geometric evolution, creating a significant disparity between semantic interpretation and physical simulation.

To bridge this gap, the authors propose HERMES++, a unified driving world model that integrates 3D scene understanding and future geometric prediction within a single framework.

Key Design Elements

BEV representation: consolidates multi-view spatial information into a structure compatible with LLMs.
LLM-enhanced world queries: facilitate knowledge transfer to the understanding branch.
Current-to-Future Link: bridges temporal gaps, conditioning geometric evolution on semantic context.
Joint geometric optimization: integrates explicit geometric constraints with implicit latent-space regularization, aligning internal representations with geometry-aware priors.

Results

Extensive validation on multiple benchmarks demonstrates the effectiveness of the approach. HERMES++ outperforms specialized methods on both future point cloud prediction and 3D scene understanding tasks.

---

*Auto-collected on 2026-05-02*