ARC-AGI-3: Why the Most Advanced AI Systems Fail in Novel Environments
> Benchmark: ARC-AGI-3 (Abstraction and Reasoning Corpus for AGI) > Designer: François Chollet (creator of Keras) > Released: March 2026 > Official site: https://arcprize.org/ | Paper: arXiv:2603.24621
Key points
- A stark gap: On ARC-AGI-3, humans score ~100% while the most advanced AI systems score below 1% on the full version — despite AI surpassing human experts on MMLU, math olympiads, and coding competitions.
- Crystallized vs. fluid intelligence: LLMs are essentially massive packages of crystallized intelligence (compressed knowledge). They excel at retrieval and pattern matching but are remarkably weak at fluid intelligence — reasoning about completely novel problems from scratch.
- The current sense of AI "generality" is largely an illusion built on crystallized intelligence — knowledge retrieval and pattern matching.
- Scaling laws have limits: model scale grew ~50,000× over five years while base LLM performance on ARC-AGI-1 stayed below 10%. Chollet: "More parameters and more data cannot solve this."
- Interactive intelligence is the next frontier — the closer benchmarks get to real-world dynamics, the weaker AI performs.
- ARC Prize 2026 carries a $2 million total prize pool, with a $700,000 Grand Prize for the first open-source system to score 100% on ARC-AGI-3.
- ARC-AGI-3 paper: arXiv:2603.24621
- ARC-AGI-2 paper: arXiv:2604.02434
- ARC Prize 2025 technical report: arXiv:2601.10904
- ARC-AGI survey: arXiv:2603.13372
- LLM fluid-intelligence analysis: arXiv:2502.07190 (Tencent WXG / HKUST)
- Chollet YC interview: https://www.ycombinator.com/library/NP-fran-ois-chollet-arc-agi-3
- ARC Prize: https://arcprize.org/
- ARC Prize 2026: https://arcprize.org/competitions/2026/arc-agi-3
- Cattell, R. B. (1963). Theory of fluid and crystallized intelligence
- Chollet (2019). "On the Measure of Intelligence"
The evolution of ARC benchmarks
ARC-AGI-1 (2019–2024): static grids
Simple colored-grid tasks: observe 3–5 input-output examples, infer the hidden rule, apply it to a new input. After years below 20%, the 2024 ARC Prize pushed open-source systems to 53.5%, then OpenAI's o1/o3 reached 87% with high-compute test-time reasoning, and by early 2026 Claude Opus 4.6 hit 93% and Gemini 3 Deep Think 96%.
Chollet's warning: These scores largely come from test-time reasoning (compute-brute-force search, not genuine understanding) and possible benchmark contamination. Base LLMs without reasoning extensions still score under 10%.
ARC-AGI-2 (2025): compositional cliff
More complex tasks requiring multi-step reasoning chains, context-dependent symbols, and symbolic interpretation:
| Model | ARC-AGI-1 | ARC-AGI-2 | Drop | |:---|:---:|:---:|:---:| | Gemini 3 Deep Think | 96.0% | 84.6% | -11.4% | | Claude Opus 4.6 | 93.0% | 68.8% | -24.2% | | GPT-5.4 Pro | ~90% | 83.3% | -6.7% | | ARC Prize 2025 winner (resource-limited) | 55.5% | 24.0% | -31.5% | | Humans | ~95% | ~100% | Nearly none |
All AI paradigms — pure neural, neuro-symbolic, program synthesis — dropped consistently by 2–3x, indicating a systematic flaw in the current AI paradigm, not an architecture-specific issue. The 2025 winner (NVARC) needed hundreds of thousands of synthetic training samples to reach 24%, showing continued dependence on knowledge coverage rather than zero-shot abstraction.
ARC-AGI-3 (March 2026): interactive environments
A paradigm shift: AI agents are placed in unfamiliar abstract worlds with no instructions and no example pairs. They must actively explore, observe feedback, build internal world models, infer goals, and plan action sequences under an action budget — mirroring how humans navigate novel real-world situations.
| | Accuracy | |:---|:---:| | Humans | ~100% | | Frontier AI systems (March 2026) | < 1% | | StochasticGoose (best so far) | 12.58% (preview version only) |
Three fundamental LLM limitations
1. Weak skill composition: The ARAOC benchmark (arXiv:2502.07190, Tencent WXG/HKUST) showed that fine-tuning on atomic operations improved them individually but did not improve composed tasks — LLMs lack genuine compositional reasoning. 2. Encoding abstract inputs: LLMs struggle to translate numeric grid matrices into visual/spatial concepts — a perception-abstraction coupling bottleneck rather than a pure reasoning failure. 3. Left-to-right decoding: Autoregressive next-token prediction is structurally at odds with the nonlinear, iterative, backtracking nature of abstract reasoning. Chain-of-Thought only simulates thinking within a linear flow.
What ARC-AGI-3 measures
Five core cognitive abilities: exploration, environment modeling, goal inference, planning under action budgets, and abstraction. Their intersection is exactly what humans use to survive unfamiliar environments — and what current AI lacks.
Benchmark contamination concerns
Evidence of contamination: during ARC-AGI-2 validation, Gemini 3's reasoning chain stated "Target is Green (3). Pattern is Magenta (6) Solid" — even though the validation prompt never mentioned ARC-AGI or the number-to-color mapping. This suggests the model memorized ARC-style tasks from training data, so high static-benchmark scores may reflect retrieval, not reasoning. ARC-AGI-3's interactive design counters this, since every episode is novel.
Possible paths forward
The most successful solutions use program synthesis and neuro-symbolic hybrids: explicitly search and compose programs from primitive operations rather than relying on neural black-box inference. NVARC's refinement loop (generate candidate programs → validate on examples → revise → repeat) partially mimics human exploratory behavior, but still depends heavily on training data and heuristic search.
Implications for the AI industry
Conclusion
Current AI is an entity with all of humanity's knowledge but lacking basic fluid reasoning. ARC-AGI-3's value is puncturing the industry's collective illusion: we may be building an enormous knowledge retrieval system rather than intelligence. True intelligence is not how much you know, but how fast you can learn when facing the unknown.