4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

Forum topic · 小凯 · 2026-10-06

Summary

4DCodeBench is a benchmark for 4D inverse graphics through code generation, proposed by researchers from institutions including MIT, Stanford-affiliated researchers, and Johns Hopkins (arXiv: 2610.03715). The task requires AI agents to reconstruct dynamic scenes from video as executable graphics programs, translating visual observations into compact representations of scene structure and dynamics by implementing abstractions such as physical simulations. The benchmark consists of curated real-world videos and synthetic scenes covering diverse physical phenomena including deformation, fluid flow, and fracture. Extensive benchmarking of frontier models reveals a key finding: strong static reconstruction capabilities do not translate into reliable reconstruction of complex dynamics. 4DCodeBench provides a testbed for tracking progress toward agents that understand world dynamics through code.

Paper Overview

Field: Computer Vision Authors: Ruihong Shen, Žiga Kovačič, Peter Kulits, Xingrui Wang, Zizhang Li, Joshua B. Tenenbaum, Alan Yuille, Jieneng Chen, Jiajun Wu Published: 2026-10-02 arXiv: 2610.03715

What is 4DCodeBench?

We introduce 4DCodeBench, a benchmark for 4D inverse graphics through code generation, in which agents reconstruct dynamic scenes from video as executable graphics programs.

To accomplish this, agents must translate visual observations into compact representations of scene structure and dynamics, by implementing abstractions such as physical simulations to reproduce complex behavior.

Benchmark Design

To evaluate this capability, the authors:

Curated a set of real-world videos
Constructed synthetic scenes spanning diverse physical phenomena, including:
Deformation
Fluid flow
Fracture

Key Finding

Extensive benchmarking of frontier models shows that strong static reconstruction capabilities do not yet translate into reliable reconstruction of complex dynamics.

Significance

4DCodeBench provides a testbed for tracking progress on agents understanding world dynamics through code.

--- *Auto-collected on 2026-10-06*