Imagine sitting for an exam where the proctor hands you not a test paper, but a pile of messy fragments: a description of a company's finances, a few blurry charts, vague email exchanges, plus your own general knowledge of how businesses operate. The question isn't "what was last year's revenue?" but:
> "Based on all of the above, decide whether this company should expand in three months, and give your reasoning plus at least two alternative options."
That's the idea behind GIM (Grounded Integration Measure) — a new kind of AI exam that the author finds genuinely exciting.
Two Old Paths: Memorization and Castles in the Air
- Knowledge deep dives (GPQA, HLE): questions deliberately obscure and hard, testing whether a model memorized some corner-case fact. This conflates "being able to search" with "being able to think."
- Pure abstract reasoning (ARC-AGI): AI dropped into a grid world of colored squares, finding transformation rules. Real reasoning, but completely disconnected from real-world context.
- Constraint satisfaction — meeting many constraints simultaneously (like writing a promise letter to your in-laws that is sincere, specific, neither too long nor too short)
- State tracking — keeping a live mental model of who said what, who lied, who was present
- Epistemic vigilance — distinguishing fact from speculation, checking source reliability
- Audience calibration — adjusting explanations for a child versus a PhD
- Single-turn only — no multi-turn interaction, mid-course correction, or iteration
- English-only — decision-making nuances may not translate across languages
- Single judge model — six or seven rubric dimensions, but judged in a closed loop with no public human baseline
Both fail to measure whether AI can integrate multiple abilities to solve real problems.
GIM's Approach: A Five-in-One Exam
A typical GIM question reads like a miniature business consulting case: a startup grew revenue 30% but margins fell 5 points, the team grew from 3 to 15, three big clients are hesitating — should the company sprint for deals or spend three months polishing the product? List your decision framework and the three most likely risks of each option.
GIM collected 820 such tasks (615 public, 205 held-out), each designed by domain experts and scored with a rubric decomposition — not right/wrong, but six or seven independent judgment dimensions aggregated into a total.
The core insight: real-world difficulty comes not from obscure knowledge, but from coordinating multiple cognitive operations at once:
A good question tests two or three at once; individually easy, together acrobatic.
What 200,000+ Exam Runs Reveal
The team tested 28 models under 47 configurations, collecting over 200,000 response-rating pairs, calibrated with a two-parameter logistic IRT model — statistically normalizing a chaotic exam with different graders and questions onto a single capability scale. This solves a longstanding evaluation problem: two 80% scores can mean very different things depending on question difficulty. IRT separates task difficulty from examinee ability.
A Restless-Making Finding: The Configuration Effect
Testing "thinking budgets" revealed that doubling a model's thinking time can improve scores more than switching to a different model. One model at default settings scored 60; with extended thinking, 72 — beating a "more advanced" model at 68.
Also notable: quantization losses vary enormously across models. Some dropped 20% at 4-bit precision, others only 5%. Deployment decisions need to consider configuration, not just raw model strength.
Nobody Gets a Perfect Score — and That's Healthy
The strongest current AI perfectly solves only ~39% of GIM tasks. Benchmarks where models hit 90–95% are measuring memory and retrieval, not real capability. A truly hard benchmark should be one no one aces — only then can it keep distinguishing good models from bad ones.
A Third Way to Measure AI Cognition
1. Knowledge benchmarks ("how much do you know") — saturating 2. Abstract reasoning benchmarks ("can you solve puzzles") — disconnected from reality 3. GIM's way: "can you integrate messy information, understand the situation, and make a grounded judgment?"
That is what we actually need from AI: clear thinking and sound decisions amid incomplete information, ambiguous environments, and stacked constraints.
Limitations
Next Time Someone Says "AI Beat Humans on Benchmark X"
Ask three questions:
1. Does it test knowledge recall or integrated reasoning? 2. Does it distinguish "remembering the answer" from "actually understanding"? 3. Can the best AI score 100% on it?
If the answer to #3 is "yes," the benchmark may be worn out. If "no" — keep digging, and you'll find AI's real capability boundary.
References
1. Patel, R., Rezende, A., & McClain, S. (2026). *GIM: Evaluating models via tasks that integrate multiple cognitive domains*. arXiv:2605.18663. Facebook Research. 2. Bubeck, S., et al. (2023). *Sparks of Artificial General Intelligence: Early experiments with GPT-4*. arXiv:2303.12712. 3. Chollet, F. (2019). *On the measure of intelligence*. arXiv:1911.01547. 4. Williams, A., Nangia, N., & Bowman, S. (2018). *A broad-coverage challenge corpus for sentence understanding through inference*. NAACL-HLT. 5. Reinforcement Learning from Human Feedback: Progress and Challenges. (2025). *Nature Machine Intelligence*, 7, 1024-1037.