SCHEMA: Teaching AI Agents to Think Like Physicists with Programmatic World Models

Forum topic · ✨步子哥 · 2026-07-19

Summary

SCHEMA is an execution framework (harness) for frontier LLM agents, presented as a one-page academic-style poster. Its core idea is a programmatic world model: rather than updating model weights, it changes the agent's reasoning process so the same frontier model performs dramatically better. SCHEMA jointly solves two tightly coupled abstraction levels—state grounding (mapping raw 64x64 pixels to objects, attributes, and relations) and mechanism discovery (learning an executable step() program that predicts state transitions from actions). Counterexamples can therefore indict either the rules or the state representation itself. The architecture runs dual loops: an outer loop (Observe → Deliberate → Execute → Record) drives interaction with the environment using an append-only timeline, while an inner loop (Theorize → Certify → Plan → Commit) plans inside the learned world model with free simulation and BFS planning. Philosophically, SCHEMA follows Einstein rather than Lorentz: when hypotheses systematically fail, the agent should redefine the state itself (e.g., inferring predicates like is_goal from interaction) instead of patching rules. The poster evaluates on the ARC-AGI-3 public set with 25 games under an RHAE metric, positioning SCHEMA as evidence that how you use a frontier model matters more than the model itself.

SCHEMA: Letting Agents Think Like Physicists

This forum post presents a one-page academic poster describing SCHEMA, an execution framework (harness) for frontier LLM agents built around a programmatic world model. Its tagline: don't change the model's weights—change its process. The same frontier model becomes qualitatively better when wrapped in this harness.

> A schema is the rule of construction. It connects abstract ideas (theories, mechanisms, predicates) to concrete percepts (pixels, trajectories, counterexamples) through a readable, verifiable, searchable program—letting an agent move back and forth between perception and concept, like a physicist at the lab bench.

Key points

Two sub-problems, solved jointly

Level 1 — State grounding: identify objects, attributes, and relations from raw pixels (observation → state program). Prior work like VIGA handles this via analysis-synthesis.
Level 2 — Mechanism discovery: learn how state changes with actions, as an executable program (s, a, s′) → step(). Prior work like WorldCoder learns transfer but requires a pre-structured starting point.
SCHEMA's contribution: encode both in the same editable program, so counterexamples can indict not only the rules but also the state representation itself.

Dual-loop architecture

Outer loop (drives game progress): Observe (64×64 pixels) → Deliberate → Execute (checked against predictions) → Record into an append-only, immutable Timeline.
Inner loop (one deliberation cycle): Theorize (write step()) → Certify (backtest) → Plan (BFS, zero-cost simulation) → Commit (commit_actions).
The outer loop records ground truth from the environment; the inner loop plans inside the model where trial-and-error is free.

Philosophical anchor: Lorentz vs. Einstein

Lorentz route: when observations contradict theory, patch the existing framework (e.g., contraction hypotheses)—keep the state, adjust parameters.
Einstein route: redefine the state itself—demote "simultaneity" from an absolute basis to part of a relative frame, and simpler electrodynamics of moving bodies emerges.
SCHEMA follows the Einstein path: when assumptions systematically fail, redefine the state; predicates like is_goal must be inferred from interaction.

Evaluation context

Benchmarked on ARC-AGI-3 (public set), 25 games, with an RHAE metric (poster dated 2026).
Framing message: *"How you use it matters more"* than which frontier model you choose.

Notes

The source is a styled HTML poster; some sections (e.g., part of the "Three Principles" section, results bars, and glossary) were truncated in the provided content, so specific performance numbers are not reproduced here.