Key points
- AVO in one line: A scaffolding layer ("harness") around Claude Opus 4.5 that lifts ARC-AGI-3 from 30% to 100% — 25 environments, 183 levels, 6,624 steps — without changing the model itself.
- Two mechanisms behind the gain:
- *Persistent memory* — keeps prior implementations, evaluations, and reasoning chains across iterations instead of rebuilding context every turn.
- *Supervisor* — a lightweight overseer that watches the main agent's search trajectory and intervenes only when progress stalls or loops appear. The main agent decides what to change, test, and submit; the supervisor only redirects when momentum dies. This division of labor lets the loop run for hundreds of cycles without derailment.
- Beyond benchmarks: Applied to GPU operator optimization, AVO ran for seven days, explored over 500 directions, submitted 40 kernel versions, and produced kernels up to 10.5% faster than FlashAttention-4 on DGX B200. The same harness, swapped to a different interface, yielded a perfect benchmark score.
- Why it matters: Long-horizon task performance is bottlenecked by scaffolding, not by the base model. Same Claude Opus 4.5, with vs. without the harness, differs by 70 percentage points. Databricks and Microsoft (April test of 19 models on long-horizon document editing) reported analogous findings.
- Competitive implication: As base models commoditize, the "architecture war" is shifting from training better models to building better harnesses — memory management, supervisor design, and failure recovery. For AI coding products (Copilot Autopilot, oncall-kit, autonomous delivery agents), the defensible moat lives in the orchestration layer, not the underlying LLM.
- Context flags: AVO is a research artifact, not an Nvidia product. Nvidia's commercial scaffolding components ship under the Nemo brand, many of them open source. ARC-AGI-3 is human-designed puzzles testing agents that must infer rules from scratch — a benchmark whose ceiling will likely shift in response.
- Project: AVO on GitHub (Agentic Variation Operators)
- Benchmark: ARC-AGI-3, 25 environments / 183 levels
- Baseline: Claude Opus 4.5 bare score 30% (highest among tested models); OpenAI models scored below 10%
- Compute footprint: 6,624 steps for the ARC run; 7-day GPU optimization run exploring 500+ directions, 40 kernel submissions
- Hardware: DGX B200, kernels compared against FlashAttention-4 (up to 10.5% speedup)
Source reference
Takeaway
The model is sufficient. The bottleneck is the harness.