The Abyss Humans Cross but AI Falls Into: What ARC-AGI Reveals About General Intelligence

> "Intelligence is the efficiency of learning new tasks from limited experience and innate priors." > —— François Chollet, 2019

This post is a deep-dive commentary on the survey *The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning* (Vahdati et al., 2026, arXiv:2603.13372), exploring what the ARC-AGI benchmark family reveals about the gap between human and machine intelligence.

What Is Fluid Intelligence and Why Might AI Lack It?

Psychologists distinguish crystallized intelligence (learned knowledge: formulas, facts, syntax) from fluid intelligence (the ability to solve never-before-seen problems). Traditional benchmarks—MNIST, ImageNet, MMLU—essentially measure crystallized intelligence: "how much do you remember?"

ARC-AGI asks a fundamentally different question:

"Given 3-5 examples, can you discover the underlying rule and apply it to a novel situation?"

The ARC-AGI Puzzles

ARC-AGI tasks are grid-based transformation problems: input is a colored grid (usually under 30×30, up to 10 colors), output is another grid. From 3-5 input-output pairs, the system must infer the transformation rule and apply it to a new test input. Each puzzle has a *different* rule—"turn blue to red" today, "rotate 90 degrees" tomorrow—and no training data covers the test rules.

Chollet deliberately grounded the benchmark in developmental psychology's Core Knowledge Priors:

1. Object permanence 2. Goal-directedness 3. Basic geometry 4. Numerosity

The Performance Cliff: ARC-AGI-1 to ARC-AGI-3

The survey's most striking finding: all AI methods drop 2-3x in performance moving from ARC-AGI-1 to ARC-AGI-2, regardless of approach (program synthesis, neuro-symbolic, or pure neural).

ARC-AGI-1 (2019):

• Accuracy stayed below 20% for years
• 2024: OpenAI's o3 first broke the 80% threshold
• 2025: Gemini 3 Deep Think reached 96.0%; Opus 4.6 hit 93.0% ($1.88/task); GPT-5.2 Pro hit 90.5% ($11.64/task)

ARC-AGI-2 (March 2025):

• Gemini 3 Deep Think: 84.6% (-11.4 points)
• Opus 4.6: 68.8% (-24.2 points)
• Kaggle resource-constrained track: best system (NVARC) only 24.03%; ARChitects 16.53%; MindsAI 12.64%

Humans: near-perfect accuracy on both versions. Expert panels scored 100% on ARC-AGI-2; average individual testers averaged 60%, with ~75% of attempters solving each puzzle.

ARC-AGI-3 Preview (July 2025) changed the game entirely: instead of static input-output grids, it presents interactive mini-games requiring exploration, goal discovery, and trial-and-error learning. Results:

• Best AI system (StochasticGoose): only 12.58% action efficiency
• Over 1,200 human players completed 3,900+ game levels
• The human-AI gap widened 8x compared to ARC-AGI-1

The 390x Cost Paradox

In 2024, o3's breakthrough on ARC-AGI-1 cost an average of $4,500 per task. A year later, GPT-5.2 Pro achieved similar performance at ~$12/task—a 390x reduction.

But the survey notes this came mainly from reduced test-time parallelism (fewer guess attempts), not deeper reasoning efficiency. Even at $12/task, AI remains 20-40x more expensive than the cognitive cost of a human solving the same puzzle.

Refinement Is Intelligence

Top-performing systems share one trait: Iterative Refinement Loops—explore multiple candidates, verify via feedback signals, adjust, and repeat until convergence.

Two 2025 ARC Prize paper award winners are especially notable:

• Tiny Recursive Model (TRM): only 7 million parameters, achieving 45% on ARC-AGI-1 via recursive latent refinement—beating far larger models.
• CompressARC: only 76K parameters, achieving 20-34% on ARC-AGI-1 using minimum description length (MDL)-based compression.

Both demonstrate that test-time training enables more efficient reasoning than massive pretraining—echoing Chollet's hypothesis that intelligence is skill-acquisition efficiency, not knowledge volume.

The Compositional Generalization Abyss

Why did ARC-AGI-2 cause such a cliff? Compositional reasoning. ARC-AGI-2 adds four difficulty axes:

1. Deeper compositional reasoning: multi-step transformations where step N depends on step N-1's state 2. Richer symbolic interpretation: context-dependent rule application 3. Control-flow complexity: if-then-else logic akin to programming 4. Resistance to brute-force search: deliberately expanded search spaces

Humans excel at compositional generalization—flexibly combining basic skills for novel situations. Current AI systems perform well on individual skills but collapse when skills must be dynamically combined.

Are We Even Measuring Intelligence?

The survey raises a philosophical question: frontier models achieve PhD-level performance in specific domains yet fail children's puzzles—suggesting reasoning remains entangled with domain-specific knowledge rather than emerging as transferable capability. Models may be retrieving memorized solution patterns rather than reasoning.

> "When frontier models hold 85% accuracy across ARC versions at $7-14 per task, while compute-constrained systems fall to 16-24% on ARC-AGI-2—are we closing the reasoning gap, or masking it with scale?"

Three Core Principles from the Survey

1. The compositional cliff persists but is narrowing. Frontier models closed some of the v1-to-v2 gap, but the 30-60 point spreads (especially in compute-constrained settings: NVARC 24%, ARChitects 16.5%) suggest progress reflects both architectural innovation and raw compute investment.

2. Accuracy converges; efficiency and transparency diverge. Leaderboard leaders approach human baselines, but as proprietary models, their training corpora, synthetic data pipelines, and potential benchmark exposure remain opaque. Progress may reflect synthetic-data coverage more than genuine reasoning.

3. Static evaluation is insufficient. ARC-AGI-3's interactive format (full release with 1000+ levels and 150+ environments coming March 2026) exposes capabilities static benchmarks cannot measure.

Conclusion

ARC-AGI reveals an uncomfortable truth: current AI's knowledge is vast, but its "imagination"—the ability to create abstractions from limited experience—is strikingly impoverished. One form of intelligence learns patterns from data and repeats success on similar patterns; the other creates understanding from sparse clues and ventures into the unknown. The former is today's AI strength; the latter is the road to AGI. The abyss humans always cross and AI still falls into may be the very essence of intelligence.

References

1. Vahdati, S., Aioanei, A., Suresh, H., & Lehmann, J. (2026). The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning. arXiv:2603.13372. 2. Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547. 3. Chollet, F., et al. (2025). ARC-AGI-2: Evaluating Compositional Generalization. ARC Prize Foundation. 4. ARC Prize Foundation (2025a, 2025b). ARC Prize 2024/2025 Competition Results. 5. Jolicoeur-Martineau, A. (2025). Tiny Recursive Model (TRM). ARC Prize Paper Award. 6. Liao, Y., & Gu, C. (2025). CompressARC: MDL-based Compression for ARC. ARC Prize Paper Award. 7. Kamradt, J. (2025). ARC-AGI-3: Interactive Intelligence Evaluation. 8. Spelke, E. S., & Kinzler, K. D. (2007). Core knowledge. Developmental Science, 10(1), 89-96. 9. Raven, J. C. (1938). Progressive Matrices: A Perceptual Test of Intelligence. HK Lewis.