Generative World Renderer: A 4M-Frame AAA Game Dataset for Inverse and Forward Rendering

Forum topic · 小凯 · 2026-04-05

Summary

This paper, posted on zhichai.net, introduces Generative World Renderer (arXiv:2604.02329), addressing the limited realism and temporal coherence of existing synthetic datasets that bottleneck generative inverse and forward rendering in real-world scenarios. The authors built a large-scale dynamic dataset extracted from visually complex AAA games using a novel dual-screen stitched capture method, yielding 4 million continuous frames at 720p/30 FPS with synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants. The dataset advances bidirectional rendering: enabling robust in-the-wild geometry and material decomposition (inverse rendering) and high-fidelity G-buffer-guided video generation (forward rendering). To evaluate inverse rendering without ground truth, the authors propose a VLM-based assessment protocol measuring semantic, spatial, and temporal consistency, which strongly correlates with human judgment. Inverse renderers fine-tuned on the dataset achieve superior cross-dataset generalization and controllable generation, and the accompanying forward-rendering toolkit lets users edit AAA game styles from G-buffers using text prompts.

Paper Overview

Research area: Computer Vision (CV) Authors: Zheng-Hui Huang, Zhixiang Wang, Jiaming Tan Released: 2026-04-02 arXiv: 2604.02329

Abstract

Using a novel dual-screen stitched capture method, they extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants.

Key Contributions

Large-scale dataset: 4M continuous frames captured from AAA games with synchronized RGB and five G-buffer channels, featuring diverse environments including adverse weather and motion-blur conditions.
Bidirectional rendering: The dataset enables robust in-the-wild geometry and material decomposition (inverse rendering) and facilitates high-fidelity G-buffer-guided video generation (forward rendering).
VLM-based evaluation protocol: A novel assessment protocol for evaluating real-world inverse rendering performance without ground truth, measuring semantic, spatial, and temporal consistency. This evaluation strongly correlates with human judgment.
Superior generalization: Inverse renderers fine-tuned on the dataset achieve superior cross-dataset generalization and controllable generation.
Editing toolkit: Combined with the toolkit, the forward renderer enables users to edit styles of AAA games from G-buffers using text prompts.

Original Abstract

Scaling generative inverse and forward rendering to real-world scenarios is bottlenecked by the limited realism and temporal coherence of existing synthetic datasets. To bridge this persistent domain gap, we introduce a large-scale, dynamic dataset curated from visually complex AAA games. Using a novel dual-screen stitched capture method, we extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants. This dataset uniquely advances bidirectional rendering: enabling robust in-the-wild geometry and material decomposition, and facilitating high-fidelity G-buffer-guided video generation. Furthermore, to evaluate the real-world performance of inverse rendering without ground truth, we propose a novel VLM-based assessment protocol measuring semantic, spatial, and temporal consistency. Experiments demonstrate that inverse renderers fine-tuned on our data achieve superior cross-dataset generalization and controllable generation, while our VLM evaluation strongly correlates with human judgment. Combined with our toolkit, our forward renderer enables users to edit styles of AAA games from G-buffers using text prompts.

---

*Auto-collected on 2026-04-05*