Events, Not Pixels: Where World Models Get the Return Wrong — Analysis and Commentary

Forum topic · 小凯 · 2026-10-22

Summary

This zhichai.net commentary dissects the paper 'Events, Not Pixels: Where World Models Get the Return Wrong' (arXiv: 2610.29435) by Liu and Tang, which examines why high-fidelity pixel-level world models (DIAMOND, IRIS, Simulus) can still mislead imagination-based planners. Using a controlled transition-level substitution method — swapping model-imagined frames with real simulator frames under identical contexts and action sequences — the study shows that 2–3 swapped frames around reward events remove a median 70% of return error, while random substitutions yield far less. Across 44 validated model-game pairs, event patches beat random controls in 39 cases (sign-test p=1.4e-7, median lift +44 points). Strikingly, event frames exhibit average or better-than-average pixel quality, and selecting frames by MSE/SSIM/LPIPS performs worse than random selection (negative lift). Most missed events are absent rather than temporally displaced. A pre-registered rule — replacing MSE prediction heads with diffusion models when baseline event recall is below 0.50 — succeeded in all 4 test cases, and repairing event frames improved planner pairwise accuracy and cut top-1 regret from 0.565 to 0.141. The piece argues evaluation should shift from rendering quality to return-error localization.

Events, Not Pixels: Where World Models Get the Return Wrong — A Close Reading

This is the second installment of zhichai.net's "Texture of Evidence" trilogy, interpreting *Events, Not Pixels: Where World Models Get the Return Wrong* (arXiv: 2610.29435) by Haoyue Liu and Xiaoying Tang (CUHK-Shenzhen, FNii-Shenzhen). Its theme: the *container* of evidence. World models render every frame beautifully — but the frames that decide decisions, the ones carrying "score" and "termination," may be empty. Clarity is not value.

Key points

Imagination-based planning (from Ha & Schmidhuber 2018 through Dreamer and MuZero) lets agents roll out candidate trajectories in a learned model, rank them by imagined return, and act on the best. In a 60-step trajectory, only a handful of transitions are events — nonzero reward or episode termination — yet the planner's entire ledger rides on them.
The proxy problem: evaluation of released world models (DIAMOND, IRIS, Simulus) leans on rendering metrics (MSE, SSIM, LPIPS). But planners don't collect pictures; they spend return. A beautifully rendered rollout can silently omit the one frame that scores.

The experiment: a surgical design

For each imagined rollout, a "parallel universe" trajectory is generated from the same context with the same action sequence, but frames drawn by the real Atari simulator. Context and actions are locked so that *who drew the frames* is the only variable.
A frozen, memoryless per-game detector (event-F1 ≥ 0.90 threshold) re-reads reward/termination from the frames, replacing the model's own bookkeeping.
Transition-level substitution with four patch types: event patches (with ±1 expansion), random patches (budget-matched control), complement patches, and full-real patches. The headline metric lift = R(event) − R(random), subtracting what blind swapping would achieve by luck.
Testbed: 26 Atari-100k games, 64×64 frames, frameskip 4, planning horizon H=60, γ=0.99; 44 reliable model-game pairs counted, plus a pre-registered "no-effect" case (Kangaroo).

Findings

Errors concentrate in event frames. On DIAMOND/Freeway, swapping just 2.8 real event steps back removes 69% of return error (random: −4%). Event patches beat random controls in 39/44 pairs (sign-test p=1.4×10⁻⁷), median lift +44 points. Frostbite lift reaches +238; Seaquest shows one honest negative (−33 for Simulus), left in the table.
Event frames look fine. Median pixel-error ratio of event frames vs. others is 0.85 — slightly *better* than average. Selecting frames to swap by MSE/SSIM/LPIPS yields negative lift (−1.1/−1.2/−0.8), hitting real events only 15–19% of the time. Pixel-metric triage loses to a coin flip.
Failure modes: 33 event-dominated, 5 non-event-dominated, 6 unclassified. DIAMOND on Boxing *hallucinates* score events (imagined return 6.67 vs. real 2.31).
Absence, not delay. Widening temporal tolerance τ barely recovers missed events (Freeway MSE-baseline recall 0.091 → 0.153 at τ=8); at least 76% of the recall gap remains — events were never drawn, just drawn late nowhere.

The pre-registered fix

Rule written before results: baseline event recall < 0.50 → a diffusion model should significantly reduce return error. Fix: swap the deterministic MSE prediction head for a DDPM (Ho et al., 2020) under identical data/steps/seeds.
4/4 correct. Freeway ΔE=+0.4503 (9× the floor); Pong +0.1560, Breakout +0.5316, Seaquest +0.7802. Ablations show the benefit comes from the architecture's distributional representation of futures, not sampling stochasticity.

Does it help planning?

From 150 real simulator states with 8 candidate plans each, repairing event frames improved pairwise accuracy in all 15 reliable pairs; median top-1 regret dropped from 0.565 to 0.141. On Pong, the diffusion model pushed pairwise accuracy from 0.5000 to 0.6110 (random-weighted control: 0.5007).
Generality: MiniGrid, Crafter, iVideoGPT on robosuite, and sparse-reward cartpole (+54/+54 lift) all replicate the concentration — across three model families, dozens of games, robot arms, and grid worlds.

Takeaway

The paper proposes no new model and tops no leaderboard. It moves the question from *"does this frame look real?"* to *"which step corrupted the return?"* Practitioners can diagnose released models as black boxes via rendered frames alone; the pre-registered recall rule is a free decision tool (recall < 0.50 → seriously consider diffusion architectures). The authors suggest a model-card-style return-error localization report alongside pixel metrics. Honest limits: returns are read via an F1-thresholded detector proxy, terminations carry only 0.7% of error, and longer horizons remain unexplored; concurrent work independently found vanishing-ball pathologies in frozen Pong world models.

Closing image: don't admire the marble first — check whether the ownership certificate's stamp pages are actually stamped. Decoration scores never measure that hole, and for years the field only read decoration scores.

References

Liu, H., & Tang, X. (2026). *Events, Not Pixels: Where World Models Get the Return Wrong*. arXiv: 2610.29435. https://arxiv.org/abs/2610.29435
Ha & Schmidhuber (2018), NeurIPS; Hafner et al. (2019, 2020); Schrittwieser et al. (2020), *Nature* 588; Alonso et al. (2024) DIAMOND; Micheli et al. (2023) IRIS; Cohen et al. (2025) Simulus (arXiv:2502.11537); Ho et al. (2020) DDPM.

Part one of the trilogy: https://zhichai.net/t/178636328