*English translation of a Chinese forum post on zhichai.net.*
> *"If you can't explain what you're doing, you may not know what you're doing at all."* — Richard Feynman
An AI in the Kitchen
Imagine hiring a self-proclaimed world-class chef to make a French soufflé. He has read every cookbook and can recite every Michelin recipe. You hand him a note with 95 simple steps: preheat the oven to 180°C, separate three eggs, mix yolks with 20 grams of flour, whip the whites, fold, and so on.
Then one of three things happens: he skips straight to step 95 and dusts sugar over raw eggs; or at step 47 he doubts whether he added flour at step 3, dumps everything, and starts over; or he stops at step 52, puts the half-finished mixture in the fridge, and declares "done."
This is exactly what today's most advanced large language models (LLMs) do.
A Carefully Designed "Honesty Test"
In May 2026, researchers at IIT Gandhinagar published an unsettling paper. Instead of testing whether AI can write poetry, code, or pass the bar exam, they asked a more fundamental question: if we give an AI an extremely simple "recipe" to follow step by step, will it actually obey?
The design was deliberately minimalist: no advanced math, no physics formulas, no subtle semantics. Only basic arithmetic programs — given two numbers, follow predefined steps to compute a result. Like testing an accountant not with consolidated financial statements but with calculator addition: if even that fails, the problem is not "capability" but the underlying execution mechanism.
They tested 14 mainstream models — from billion-parameter open-source models to hundred-billion-scale commercial giants — across 55 datasets, with program lengths from 5 to 95 steps.
The results:
- 5-step programs: 61% average first-answer accuracy.
- 95-step programs: 20%.
- Title: When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
- Authors: Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh
- Institution: Indian Institute of Technology Gandhinagar (IIT Gandhinagar), et al.
- arXiv: 2605.00817
- Published: May 1, 2026
- Field: Computation and Language (cs.CL)
- Length: 77 pages, 109 figures
- 14 mainstream LLMs tested across 55 datasets
- Program lengths: 5 to 95 steps
- 5-step average first-answer accuracy: 61%
- 95-step average first-answer accuracy: 20%
- Five failure modes: Missing Answers, Premature Answers, Self-Correction After Initial Error, Under-Executed Traces, Hallucinated Extra Steps
When you give a model claiming "general intelligence" a 95-step instruction, it has an 80% chance of going wrong somewhere — or simply giving up.
Five Ways AI "Zones Out"
The paper's most valuable contribution is a forensic dissection of model failures, which cluster into five patterned symptoms:
1. Missing Answers
The model decides a step "isn't important" and skips it. This isn't true memory loss — LLMs have no memory. Attention is like a flashlight beam: steps outside the beam simply vanish as the program grows longer.2. Premature Answers
Halfway through, the model "gets it" and blurts out a final answer, ignoring the dozens of remaining steps. It is not executing a program; it is guessing what answer looks most plausibly correct. When generated text starts resembling a reasonable final output, the probability engine says: "Stop now!"3. Self-Correction After Initial Error
The model makes a mistake, "notices" it, tries to fix it — and makes everything worse. Rather than returning to the faulty step, it patches over the current mess, like a captain ordering the crew to bail water instead of plugging the hole in the hull.4. Under-Executed Traces
The model starts well, then suddenly stalls mid-way and outputs an obviously incomplete answer. As dependencies between steps grow complex and the model must repeatedly "look back," its context window becomes a repeatedly scribbled-over draft — and it halts, often with false confidence.5. Hallucinated Extra Steps
The most ironic failure: the model executes all steps and then invents additional steps that don't exist, like an over-enthusiastic chef adding chili sauce to a recipe that never asked for it. Because LLMs are probability-driven text continuers trained to "generate plausible text" rather than "faithfully execute," a text that "looks unfinished" triggers the urge to keep writing — great for poetry, fatal for programs.Why "Smart" Doesn't Mean "Obedient"
These findings expose a fundamental trap: we've been using the wrong yardstick for AI. When GPT-4 passes the bar exam or a model solves math problems, we cheer "AI can reason!" But this study coldly shows: the accuracy of final answers masks systematic collapse during execution. A model may arrive at the correct answer through statistical shortcuts without ever truly executing the specified steps.
Humans executing multi-step programs maintain a mental model — a dynamic cognitive structure tracking where we are, intermediate results, and what comes next — supported by working memory, metacognitive monitoring, and resistance to interference. LLMs have none of this. They process one token at a time through a giant feed-forward network; "attention" merely selects keywords from context. When context grows long, selection becomes vague guessing. It's like using a weather-forecasting model for navigation: fine for "it may rain tomorrow," but by the third intersection it starts inventing street names.
A Red Light for the Agent Era
This research matters precisely when AI Agents are at their peak: booking flights, writing code, managing projects — all requiring multi-step, long-horizon execution with intermediate state. If a base model achieves only 20% accuracy on a 95-step program, how can we trust it with real-world tasks involving hundreds of steps? Imagine an agent planning a wedding that, at the 95th subtask, forgets the confirmed date or invents a nonexistent guest list.
The authors offer no simple fix — because this is not a bug but an architectural, structural flaw. A real solution may require new paradigms beyond the current Transformer architecture, or at least explicit procedural execution layers tightly coupling LLMs with symbolic reasoning engines or deterministic execution environments.
What Would Feynman Say?
Feynman liked to say: "If you think you understand something, try teaching it to a computer." By his standard, today's LLMs are excellent imitators of understanding but poor practitioners of execution. They can produce brilliant essays on quantum mechanics, but when asked to compute along fixed steps, they lose their way by step five.
This doesn't negate the revolutionary value of LLMs — indeed, the researchers used LLMs to generate test programs and evaluation data. But a tool's power depends on clear-eyed knowledge of its limits. As Feynman said after the Challenger disaster: "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."
Before handing critical tasks to AI, let's confirm it can really count to ninety-five without zoning out.
Paper Details
Key data: