Schema Harness Jumps Claude Opus 4.8 + Fable 5 from 42.83% to 98.98% on ARC-AGI-3

Forum topic · 小凯 · 2026-07-19

Summary

Schema, an open-source agent harness from schema-harness.github.io, encodes ARC-AGI-3 environments as runnable world-model programs (world_model.py) rather than vector representations. On the ARC-AGI-3 public leaderboard's 25 games, Schema + Claude Opus 4.8 + Fable 5 scores 98.98% RHAE, versus 42.83% for the same models under Claude Code — a 56.15 percentage-point controlled improvement achieved by changing only the harness, not the model. Schema + GPT-5.6 Sol reaches 95.35%. The design enforces three constraints: the world model must be an executable step(state, action) program, every recorded transition must pass backtesting before use, and actions are only sent via commit_actions with any prediction mismatch immediately aborting the remaining plan. Its inner loop is theorize (coding) → certify (backtest) → plan (BFS) → commit, jointly solving state attribution and mechanism discovery in one editable, diffable, readable program. Field-reported scores are self-reported and not yet independently verified by ARC Prize; generalization to unseen environments remains to be evaluated.

> On the ARC-AGI-3 public leaderboard, Claude Opus 4.8 + Fable 5 scored only 42.83% when orchestrated by Claude Code, but with the Schema harness layered on top the score jumped to 98.98% — an absolute gain of 56.15 percentage points. The models were unchanged; only "how the models are used" changed. GPT-5.6 Sol reaches 95.35% under the same harness. This suggests the harness has become the dominant abstraction layer of the agent era, with model weights a secondary variable.

The Results

Schema appeared on Hacker News (July 17) as a project that encodes the ARC-AGI-3 environment's world model as a "runnable program" rather than a vector representation. Paper and code are at schema-harness.github.io. RHAE scores on the 25 games of the ARC-AGI-3 Public set:

| Configuration | RHAE Score | |------|----------| | Schema Harness + Claude Opus 4.8 + Fable 5 | 98.98% | | Schema Harness + GPT-5.6 Sol xhigh + Sol max | 95.35% | | Claude Code + Opus 4.8 + Fable 5 (controlled baseline) | 42.83% | | GPT-5.6 Sol max (ARC Prize official best single variant) | 13.33% |

The controlled comparison changes only the harness, not the model: a 56.15-point improvement.

How Schema Works

Schema's core idea: the model encodes the environment as a Python program (world_model.py) instead of acting directly on the game. Three hard constraints:

1. The world model must be an executable step(state, action) program. 2. Every recorded transition must pass validation (backtesting) before use. 3. All actions go through commit_actions; any prediction error immediately discards the remaining plan.

The outer loop is observe → deliberate → execute → record. Within a single deliberation: theorize (coding) → certify (backtest) → plan (BFS) → commit. The model first writes code predicting what happens next, replays historical data to verify it, runs BFS for the shortest path, and only then commits actions.

If one step mispredicts by a single cell, Schema stops immediately and uses the error as a counterexample to revise step(). The entire world model is "a text file — diffable and readable," which is both its source of interpretability and the reason it nearly maxes out all 25 games.

Historical Progress

| Time | RHAE | |------|------| | March 2025 | 0.51% | | July 2025 (GPT-5.6 Sol max) | 7.78% | | GPT-5.6 Sol max / Public | 13.33% | | Claude Code + Opus 4.8 + Fable 5 (baseline) | 42.83% | | Schema + Opus 4.8 + Fable 5 | 98.98% |

From 0.51% to near-perfect in 18 months. Schema is the largest single jump in that curve.

Analysis

"How you use the model matters a lot" — the rise of the harness abstraction

Claude Opus 4.8 and GPT-5.6 Sol are both flagship models, yet under a general-purpose harness (Claude Code) they score 42.83% and 13.33% on ARC-AGI-3. Under a harness designed for "explicit world model + backtest verification + plan/action separation," both approach 100%. This points the way for AI applications in 2026 H2: the competitive battlefield for front-end engineering, product differentiation, and agent frameworks is no longer base models but harness orchestration. OpenAI Operator, Anthropic Claude Code, Manus, Genspark, and Devin are all, broadly, harnesses — and Schema turns this abstraction layer into readable code and a verifiable paper.

Jointly solving state attribution and mechanism discovery

VIGA and WorldCoder each attacked one side of a problem that cannot be solved independently: state attribution (which pixels correspond to objects, players, walls, counters) and mechanism discovery (how state changes when a key is pressed). A plausible state partition can be falsified by inconsistent transition rules. Schema encodes both in the same editable program: the model can revise rules while observing, or revise its state hypotheses while revising rules. The paper's analogy is special relativity: the Lorentz route (keep the ether, change the rules) versus the Einstein route (drop the ether as part of the state, change "what the state is"). Schema encourages the Einstein route — when rules can't explain a failure, the model may redefine the state itself. Beyond ARC-AGI-3, this applies to any environment with unknown rules: robot manipulation, production-line anomaly handling, long-run operations.

A real case: the LS20 level

The paper shows an iteration trace: Opus 4.8 committed [2,2,2,3,2,2] on level LS20, and the Block redrew the indicator in maroon — 77 cells mispredicted. A typical agent would retry a different action sequence. Schema instead halts the plan, writes the mismatch as a counterexample into world_model.py, revises the "which colors are the indicator" rules, backtests, then BFS-finds the shortest path. Seven levels, 642 steps vs. 780 human steps, RHAE 100%. The key design: a single prediction mismatch halts execution, closing a trial-and-revise loop at every decision granularity.

Why It Matters

1. The harness abstraction is the dominant battleground for AI agents in 2026 H2. Schema makes harness design readable, reproducible, and falsifiable. 2. Changing the harness alone moved the score by 56 points — the hardest data yet pulling industry attention from raw model iterations back to harness design. 3. Fully open source with a public paper — the first near-perfect ARC-AGI-3 result anyone with frontier-model API access can reproduce independently. 4. The paradigm transfers: explicit world model + backtest verification + plan/action separation applies to robotics, manufacturing, and operations.

Risks and Open Questions

The 98.98% is self-reported and not independently verified by ARC Prize; performance on the semi-private set is unknown.
The GPT-5.6 Sol comparison is not a strictly controlled ablation: 95.35% uses an xhigh + max fallback configuration vs. 13.33% for the official best single variant.
Results cover 25 public games; generalization to novel environments needs more evaluation.
Open-sourcing the harness abstraction is not the same as open-sourcing harness engineering — integrating it into production tools like Claude Code or Cursor is a separate challenge.

---

Sources:

Schema Harness paper and project page: https://schema-harness.github.io/
Hacker News front-page discussion (2026-07-17)
ARC-AGI-3 public leaderboard baselines: ARC Prize official public dataset