> *GPT 5.2 scores only 9.8% accuracy, Gemini 3 Pro only 6.1%. This is not a defect — it is a wake-up call.*
---
🏃 A Long-Run Test with Shocking Results
Imagine designing a 50-kilometer endurance course split into ten 5-kilometer segments. Each segment alone is easy for professional athletes. But they must run it all at once — no substitutions, no rest, no forgetting the route already covered.
Now assemble the world's top runners:
- GPT 5.2 (OpenAI): 9.8% completion rate
- Gemini 3 Pro (Google): 6.1%
- Other frontier models: all below 10%
- MATH dataset: a few thousand tokens per problem on average
- AIME competition problems: reasoning length typically under 5K tokens
- FrontierMath: capped at 10K tokens
- HLE (Humanity's Last Exam): average reasoning under 5K tokens
- Does LongCoT really measure "reasoning"? If models succeed at individual sub-problems only probabilistically, cascaded failure may reflect mere error accumulation rather than a distinct long-horizon deficit.
- Are expert-designed dependency graphs too artificial? Real-world reasoning is messy, nonlinear, and full of distractors; LongCoT's clean structure may be idealized.
- Why is GPT 5.2 better than Gemini 3 Pro? The paper doesn't deeply explain the gap — architecture, data, or post-training?
- What does 62K tokens mean? Models can already generate very long reasoning chains — but length isn't the issue; quality is.
This is the stark reality revealed by the LongCoT benchmark. The numbers don't lie: 9.8% isn't a small-numbers problem — it's a fundamental one.
🎭 Why Was This Never Noticed Before?
The Sprint Champion Illusion
Before LongCoT, AI reasoning benchmarks were essentially sprint races:
On these sprints, AI performs impressively — 60–80% accuracy. But we made a mistake: mistaking sprint champions for marathon runners.
As Feynman put it in his Challenger investigation context:
> "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."
Why Is Long-Horizon Reasoning Harder?
LongCoT doesn't simply make problems longer. It builds a dependency graph structure: each problem contains dozens of interdependent sub-questions — sub-question 2 depends on 1's answer, sub-question 5 needs results from 3 and 4, and so on.
It's not running a straight 50 km; it's navigating a maze where every turn depends on prior choices. One wrong step, and everything after fails.
As Feynman might explain:
> "It's like long division. Every step is simple — a schoolchild can do it. But over 100 steps, you must retain every intermediate result. If at step 73 you discover step 12 was wrong, you must go back. That requires not just computation, but memory + focus + backtracking + error detection."
🔬 LongCoT's Five Domains: A Carefully Designed Maze
LongCoT contains 2,500 expert-designed problems across five domains:
1️⃣ Chemistry: Reaction Cascades
Identify compound A from structure, match compound B, predict reaction product C, then react C with D to get E... up to N9, finally computing molecular diameters of multiple products. Each sub-problem is doable for a chemistry student — but an error at N1 cascades through N3, N5, and the final answer.2️⃣ Mathematics: Chained Competition Problems
14 linked high-school competition problems. N1's answer is a parameter k; N2 depends on k and branches into two paths; the chain continues to N14. Errors amplify catastrophically — like a building whose foundation is off by one degree.3️⃣ Chess: Minimax Search on a 30×30 Board
A knight and 8 pawns on a huge board. Alice maximizes total moves (knight capturing all pawns, always choosing the nearest pawn); Bob minimizes them. A game-tree search problem requiring depth-8 minimax with memoization — naively 8! = 40,320 capture orders. GPT 5.2's performance here wasn't disclosed separately, but given the overall 9.8%, likely disastrous.4️⃣ Computer Science: Dependency Graph Construction
Program execution tracing, call-chain understanding, loops and conditionals — track variables across long executions and produce correct output.5️⃣ Logic: Constraint Satisfaction
Find solutions satisfying interdependent constraint graphs.📉 Failure Mode Analysis: Why Does AI Collapse?
The paper identifies five characteristic failure modes:
1. Context degradation — beyond ~50K tokens, models begin "forgetting" earlier content. Transformer self-attention is O(n²), and even efficient variants lose information over very long sequences.
2. Plan drift — a sound initial plan gradually derails over a long reasoning chain. Like driving for two hours without checking the map. This reveals a lack of metacognition.
3. Partial results lost — the model computes an intermediate value (say A = 42), then later "recalls" it as 24, corrupting everything downstream.
4. Giving up early — after some length, models suddenly emit a (likely random) answer. A training bias learned from short benchmarks: "output an answer fast," not "persist until solved."
5. Errors undetected — mistakes go unnoticed mid-reasoning. Humans ask "does this look right?" Current AI lacks this self-monitoring. As Feynman said:
> "The first principle is that you must not fool yourself — and you are the easiest person to fool."
🧠 What Does Long-Horizon Reasoning Require?
The LongCoT authors define four core capabilities:
1. Planning, exploration, and backtracking — trying multiple paths and retreating from dead ends. 2. Long-term state maintenance — tracking all relevant information and dependencies across the chain. Humans solve this with external tools (notebooks, whiteboards); AI must hold it all in its reasoning chain. 3. Progress monitoring and error detection — the metacognitive ability to ask "is this result correct?" 4. Credit assignment — when the final answer is wrong, locating which intermediate step failed.
> "These aren't 'advanced' abilities — they're the basic constituents of reasoning. If you can't monitor your progress, detect errors, or backtrack, you're not reasoning; you're mechanically executing steps."
📊 The Data
| Model | Accuracy | Avg. reasoning tokens | |-------|----------|----------------------| | GPT 5.2 | 9.83% | 62,000 | | Gemini 3 Pro | ~6.1% | — | | Other frontier models | <10% | — |
Pass@K analysis: even allowing multiple attempts, improvement is marginal — the problem is systematic incapacity, not occasional error.
With code execution: helps in programmatic domains, but for compositional reasoning (math, chemistry) performance remains near zero. Tools assist computation; they cannot substitute for long-horizon reasoning itself.
🎓 Implications for AI Research
1. Redefining "reasoning" — short-horizon and long-horizon reasoning may be entirely different capabilities, like sprinting vs. marathon running. 2. Limits of test-time scaling — GPT 5.2 used 62K tokens for 9.8% accuracy. If you can't manage a long reasoning chain, more tokens just amplify errors. The issue is not thinking longer but thinking better. 3. Architectural redesign — quadratic attention complexity, no explicit state management, no built-in backtracking or error detection. Real breakthroughs may need new architectures, not just bigger models. 4. Training data — current corpora are dominated by short texts; we may need to construct long-horizon reasoning training data, not just benchmarks.
🔮 Critical Questions
🌌 A Broader Perspective
Humans excel at long-horizon reasoning because of extended working memory (paper, re-reading), metacognition, error monitoring, and world models. These capacities evolved under selection pressure — planning hunts, tool-making, social tracking. Current AI training objectives (next-token prediction, reward maximization) may simply not produce them. What's needed is selection pressure during training, not just at test time.
📝 Conclusion: A Lighthouse Over the Abyss
9.8% accuracy is not a failure — it is a discovery. It tells us how far we are from genuine intelligence. This is not a problem solvable with "more data" or "bigger models"; it demands new ideas, architectures, and training paradigms. And precisely because it is hard, it is worth solving.
> "It is much more interesting to live not knowing than to have answers which might be wrong."
Admitting we don't know how to make AI reason over long horizons is far better than pretending we've solved it. That is scientific honesty — and the continuation of the Feynman spirit.
---
📚 References
1. Motwani, S. R., Nichols, D., London, C., et al. (2026). LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning. *arXiv preprint arXiv:2604.14140*. 2. Feynman, R. P. (1974). Cargo Cult Science. *Caltech Commencement Address*. 3. Hendrycks, D., et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. *NeurIPS*. 4. Glazer, E., et al. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. *arXiv preprint*. 5. Phan, L., et al. (2025). Humanity's Last Exam. *arXiv preprint*. 6. Feynman, R. P. (1985). *Surely You're Joking, Mr. Feynman!* W. W. Norton & Company. 7. Feynman, R. P. (1999). *The Pleasure of Finding Things Out*. Perseus Books.