VISTA: A Visual Harness for Reasoning in an Interactive World — Structured English Overview
Source: a long-form Chinese forum analysis (framed in the style of Richard Feynman) of the paper *VISTA: A Visual Harness for Reasoning in an Interactive World* (arXiv:2610.02200v1) by Qiushi Han, Keya Hu, and Linlu Qiu — co-first authors in the MIT teams of Kaiming He and Cathy Wu. Open-sourced at https://github.com/vista-research/vista.
Key points
- The problem: ARC-AGI-3 (Abstraction and Reasoning Corpus for AGI, 3rd generation; by François Chollet et al.) consists of 25 multi-level interactive 2D grid games (183 levels total). Agents must explore unknown physics via actions (
ACTION1–ACTION7) with no prior rules, rewards, or object semantics. - What previous top systems did: Leaderboard systems (Schema, Tycho, Retrodict) flattened the 64×64 color grid into 4,096 decimal integers as text (~4,000 tokens/frame) and performed "explicit executable world-model synthesis" — writing up to 4,055 lines of Python (71 functions) per game and burning ~71.9M tokens/game on code generation and debugging. Under the official text baselines, GPT-5.6 Sol scored only 13.33 RHAE, Claude Opus 5.0 40.68, Gemini 3.1 Pro 10.68 (human = 100).
- What VISTA does (zero trainable parameters, three mechanical components): 1. Direct visual observation: renders grids to 512×512 PNG via 8× nearest-neighbor upsampling (no edge blending), drawing 1px grid lines by adaptively darkening pixels 25% —
- Modern VLMs suffer one-shot lossy visual encoding: 4,096 grid cells compressed into ~308 tokens causes high-frequency aliasing (small key objects lose their coordinates/colors), and compaction destroys or text-summarizes old image tokens — the agent can never "rewind the tape." VISTA's thesis: the missing piece is not reasoning ability but a mechanism to preserve old frames losslessly and re-attend with a magnifying glass when new questions arise.
- The post (channeling Feynman's "Cargo Cult Science") argues code-synthesis world models force premature formalization and brittle retrodiction: one new object color crashes thousands of lines of code, trapping agents in bug-fixing loops instead of exploring.
- ARC-AGI-3 3D isometric rendering: 84.12 RHAE (symbolic code-synthesis systems fail on this rendering entirely).
- GameWorld (34 games, 170 tasks): success rate 63.33% ± 2.37%, progress 79.30% ± 2.14%.
- AI GameStore (10 commercial HTML5 games: 2048, Minesweeper, Pac-Man, Tetris, etc.): normalized scores 119.99 / 148.58 / 140.06 across three runs (human median = 100).
- BabyVision (39 static fine-grained maze/curve-tracing puzzles): accuracy 63.23% ± 3.00% (best 66.7%) vs. 41.0% one-shot GPT-5.6 Sol baseline.
(pixels[mask] * 3) // 4 — rather than fixed gray/black lines that would vanish or create false edges against the 16-color palette. Result: ~308 visual tokens/frame (77 at 256×256), a 10–50× efficiency gain that preserves 2D topological adjacency.
2. An 87-word prompt implementing scientific method: *predict the visual change before each action; state the actual change after; build and revise a compact game model in GUIDE.md; prefer the simplest hypothesis consistent with evidence; test uncertain parts with targeted actions.* Supported by dual-file memory: GUIDE.md (64KB cap, durable cross-level laws) and WORKING.md (16KB cap, current-level scratchpad, auto-reset on level change).
3. Lossless visual memory + explicit re-attention: every frame (including intermediate animation frames) is archived as a read-only PNG (os.chmod(path, 0o400)); by default only the terminal frame enters context. Two tools let the agent re-look: inspect (up to 16 ROI crops per call, nearest-neighbor upscaled to 512px, red-box context on some backends) and read_pixels (up to 64 views / 4,096 samples, equal-bin-center sampling \(x_c = x + \lfloor (2c+1)w/(2C) \rfloor\) deliberately avoiding the darkened grid lines, colors mapped to a 64-char symbol table with legend).Why this works (the post's analysis)
Scoring metric
RHAE (Relative Human Action Efficiency): per-level score \(S_{\ell,g} = \min(1.15, (h_{\ell,g}/a_{\ell,g})^2)\) — squared penalty for inefficiency; per-game score is the min of weighted completion rate and weighted efficiency with \(w_\ell = \ell\), capped at 100.
Main results (ARC-AGI-3 public set)
| Agent | RHAE | Notes |
|---|---|---|
| Human reference | 100.00 | 183/183 levels, 17,135 steps |
| Gemini 3.1 Pro baseline | 10.68 | text grids |
| GPT-5.6 Sol baseline | 13.33 | text grids |
| Claude Opus 5.0 baseline | 40.68 | text grids |
| GLM-5.3 Flash (320B open weights) + VISTA | 66.93 | outperforms baseline closed frontier models |
| GPT-5.6 Sol + VISTA | 99.00 (3-run mean \(99.04 \pm 0.17\)) | 182/183 levels, 9,126 steps (46.7% fewer than human); missed only ls20 level 7 |
| Claude Opus 5.0 + VISTA | 100.00 | 183/183 levels, 7,302 steps — 57.4% fewer than human; per-game step savings on all 25 games |
Token consumption dropped to ~30.7M per game (57.3% savings vs. code synthesis).
Ablation (GPT-5.6 Sol, additive stages)
| Stage | Addition | RHAE | Δ |
|---|---|---|---|
| Baseline | official text baseline | 13.33 | — |
| i | 2D PNG input with grid lines | 47.32 | +33.99 |
| ii | 2,000-step / 48h budget | 51.66 | +4.34 |
| iii | multi-turn continuous dialogue | 65.82 | +14.16 |
| iv | dual notes + 87-word guidance | 70.05 | +4.23 |
| v | lossless visual archive + inspect | 94.10 | +24.05 |
| vi | read_pixels (full VISTA) | 99.00 | +4.90 |
The two "rockets": image input (+33.99) and lossless visual memory with re-attention (+28.95 combined). A case study (vc33, level 6) illustrates how inspect/read_pixels unblocks levels that stall without re-look tools.
Generalization (same harness, unchanged core)
Takeaway
The post's central claim: "look closely on demand and take notes while looking" is not a game-specific trick but the universal skeleton of fine-grained visual cognition. VISTA achieves frontier-agent dominance on ARC-AGI-3 with no new parameters, no neural modules, and no world-model code — only better observation formatting, minimal note-taking scaffolds, and deterministic re-inspection tools.
*Note: the source post was truncated near the vc33 case study; details after that point are not included here.*