Equations Written Correctly, Yet the Water Won't Flow: Testing AI's Physical Intuition with WorldSolver

Forum topic · 小凯 · 2026-10-07

Summary

This forum post reviews WorldSolver, a benchmark from the Chinese Academy of Sciences' Institute of Automation, Peking University, and Huawei Noah's Ark Lab that measures whether large language models truly understand physics by asking them to implement working physics simulators. Built from 168 simulation tasks extracted from 61 classic computer graphics papers across seven domains—fluids, rigid bodies, plastic materials, elastic solids, rods and strands, cloth and shells, and multi-physics coupling—the benchmark isolates solver implementation from scaffolding. Scoring combines execution, visual realism judged by a VLM, and quantitative physics-law violation checks. Seven frontier models were tested: GPT-5.6-Sol (48.7) and Claude-Opus-5 (46.7) lead, but all fall far below 100, with a steep 18-point gap to the rest (Gemini-3.7-Flash at 29.3, Kimi-K2.7-Code at 17.1). Although submission rates were high (88–100%), valid trajectories dropped sharply and pass rates fell to 8–41%, including failures on classic setups like Newton's cradle. Behavioral analysis shows edit-tool usage correlates with success more than attempt count. The conclusion: current models largely memorize physics rather than embody intuitive understanding.

Overview

This post is part of a series on AI embodiment, focusing on the most primitive and often overlooked sense: physical intuition. Humans develop it through bodily experience—babies as young as 18 months know how eggs crack or how a Newton's cradle behaves, without solving any equations. LLMs, by contrast, have read every physics textbook yet have never been splashed by water or carried a swaying shopping bag. Do they *understand* physics, or just recite it?

The paper under review, WorldSolver, comes from the Institute of Automation (CASIA), Peking University, and Huawei Noah's Ark Lab.

The Benchmark

168 simulation tasks distilled from 61 classic computer graphics papers, covering 7 physical domains: fluids, rigid bodies, plasticity/complex materials, elastic bodies, rods/strands/chains, cloth/shells, and multi-physics coupling.
Each task provides a fixed code scaffold S = (G, P, E, R): scene geometry, physical parameters, execution protocol, and rendering config. The model's only job is to implement the solver Φ, where the dynamics follow:

s_{t+1} = Φ(s_t)

It is the first benchmark to isolate solver implementation itself—prior work either asked models to build everything from scratch (confounding scaffold differences) or scored only numbers/video in isolation, or tested a single domain.

Three-Tier Grading

1. Execution (E): does the code run? Failure zeroes the score. 2. Visual realism (V): a VLM judge (GPT-5.6-Sol) scores 20 sampled frames on event sequence, interaction response, signature phenomena (vortices, ripples, catenary curves), and temporal coherence (0–1 each). 3. Physical plausibility (P): 3–5 predefined violation functions per task, checked numerically—mass conservation, momentum/impulse consistency, interpenetration.

Final score: s_k = (V_k + P_k) / 2, only if execution passes. This dual track defeats both fake-but-runnable code and visually stiff-but-numerically-passable simulations.

Main Results

Seven frontier models competed (max reasoning effort, 1800s generation / 600s runtime per task):

| Model | Overall | |---|---| | GPT-5.6-Sol | 48.7 | | Claude-Opus-5 | 46.7 | | Gemini-3.7-Flash | 29.3 | | DeepSeek-V4.1-Flash | 24.6 | | GLM-5.3 | 23.6 | | Qwen-3.8-Max | 22.7 | | Kimi-K2.7-Code | 17.1 |

Even the best score only about half. An 18-point cliff separates the top two from the pack.
Skill profiles differ sharply: GPT-5.6-Sol leads fluids (49.6), rigid bodies (54.8), rods/strands (59.5), and multi-physics (51.5); Claude-Opus-5 dominates plasticity (47.9), elastomers (61.9), and cloth/shells (60.1). In rods/strands GPT leads by 26%; in cloth Claude leads by 24.3%. Physical intuition is not one organ but a set of separately trained muscles.
Worst cases: Gemini scores 9.5 on cloth; GLM-5.3 just 7.0 on multi-physics.

The Three-Layer Funnel

Solver Submitted: 88–100% for all models.
Trajectory Produced: drops to 82% for GPT, 38% for Kimi—most code that looks plausible crashes or produces non-physical trajectories.
Cases Passed (≥60): GPT 41%, Claude 37%, Qwen 14%, Kimi 8%.

Kimi's 99% submission vs. 38% valid trajectories is a collapse of confidence: syntactically flawless code, physically broken output. Visual examples include buoyant smoke that vanishes mid-rise (Kimi), dam-break water exploding into particles instead of flowing (Qwen), and a Newton's cradle whose middle balls scatter (Kimi).

Behavioral Insights

1. More attempts ≠ better: GPT averaged 13.07 epochs (lowest) yet scored highest; Kimi averaged 37.95 and scored lowest. 2. Editing beats writing: Edit-tool usage—Claude 25.3%, GPT 12.5%, Gemini 8.2%, Kimi 2.5%—ranks almost identically to total score. Self-skepticism and rework are the watershed. 3. Cost: GPT averaged 8.89 min/task; Claude 18.59 min for nearly equal score; Gemini was fastest (2.59 min) but weak. DeepSeek-V4.1-Flash offered the best score-per-compute.

Conclusion

The verdict: current models mostly recite physics rather than embody it. Everyone submits; most fail to run; fewer than half pass; even textbook Newton's cradles break. WorldSolver turns the vague notion of "physical intuition" into 168 concrete tasks and quantitative diagnostics. True physical intuition, in humans, comes from a body that gets bumped and splashed—if AI is ever to gain one, a simulator it builds and refines may be its first layer of skin.

References

Paper: *WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?*
Authors: Siru Jiang, Yongzhe Lyu, Shuo Lu, Yubin Wang, Yuxiang Zhang, Yue Liao, Bin Wang, Jian Liang, Tieniu Tan
arXiv: 2610.08720 (v1, 2026-10-06)
Institutions: NLPR & MAIS, CASIA; Peking University; Huawei Noah's Ark Lab
Code: https://github.com/sirujiang/WorldSolver