Paper Overview
Research Area: Computer Vision Authors: Zheng-Hui Huang, Zhixiang Wang, Jiaming Tan Published: 2025-04-01 arXiv: 2504.01263
Summary
Scaling generative inverse and forward rendering to real-world scenarios is bottlenecked by the limited realism and temporal coherence of existing synthetic datasets. To bridge this persistent domain gap, the authors introduce a large-scale, dynamic dataset curated from visually complex AAA games.
Using a novel dual-screen stitched capture method, they extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants.
Key Contributions
- Large-scale dynamic dataset: 4M continuous frames (720p/30 FPS) with synchronized RGB and five G-buffer channels, captured via dual-screen stitching from AAA games across diverse conditions.
- Bidirectional rendering: Enables robust in-the-wild geometry and material decomposition (inverse rendering) and facilitates high-fidelity G-buffer-guided video generation (forward rendering).
- VLM-based evaluation protocol: A novel protocol measuring semantic, spatial, and temporal consistency to evaluate real-world inverse rendering performance without ground-truth annotations.
- arXiv: <https://arxiv.org/abs/2504.01263>
Findings
Experiments demonstrate that inverse renderers fine-tuned on this dataset achieve superior cross-dataset generalization and controllable generation, while the proposed VLM-based evaluation correlates highly with human judgment.
Combined with the released toolkit, the forward renderer lets users restyle AAA game footage using text prompts through G-buffer editing.