AI's Amnesia: Why Frontier LLMs Fall Apart on 95-Step Instructions

> *"If you can't explain what you're doing, you probably don't know what you're doing."* — Richard Feynman

The AI Chef in the Kitchen

Imagine hiring a "world-class" chef to make a soufflé. He has read every cookbook, can recite every Michelin recipe. You hand him a note with 95 simple steps, from preheating the oven to dusting powdered sugar. Then:

• He jumps straight to step 95 and sprinkles sugar on raw eggs.
• Or at step 47, he doubts whether he added flour at step 3, dumps everything, and starts over.
• Or he stops at step 52, puts the half-finished mess in the fridge, and declares "done."

This is exactly what today's most advanced large language models are doing.

An Honest Stress Test

In May 2026, a research group at IIT Gandhinagar published a striking diagnostic study. Instead of testing whether LLMs can write poetry or pass the bar exam, they asked a sharper question: if you give an AI a very simple 'recipe' and ask it to follow the steps, does it actually obey?

The design was deliberately reductive: no advanced math, no physics, no subtle semantics—only basic arithmetic organized into procedural programs. They evaluated 14 mainstream models (from small open-source models to trillion-parameter commercial giants) across 55 datasets, with program lengths from 5 to 95 steps.

The results:

• 5-step programs: 61% average first-answer accuracy
• 95-step programs: 20% accuracy

In other words, given a 95-step instruction, a frontier model has an 80% chance of failing somewhere—or simply giving up.

Five Ways the AI "Zones Out"

The paper's most valuable contribution is a forensic taxonomy of failure modes:

1. Missing Answers

The model simply skips steps, as if they don't matter. This isn't real memory loss—LLMs have no memory. Attention is like a flashlight beam: when the program is long enough, steps fall outside the beam and "disappear."

2. Premature Answers

Halfway through, the model decides it "gets it" and outputs a final answer, ignoring the dozens of remaining steps. This exposes a deep bias: the model isn't executing a program—it's guessing what answer looks most plausible, and wraps up as soon as the text starts resembling a reasonable output.

3. Self-Correction After Initial Error

The model makes a mistake, "notices" it, tries to fix it—and makes everything worse. Rather than returning to the erroneous step, it keeps patching on top of a corrupted state, like a captain ordering the crew to bail water while the hull leaks.

4. Under-Executed Traces

The model starts well, runs out of steam midway, and produces an obviously incomplete answer—often with a false sense of confidence, like a student handing in half an essay. As dependencies between steps pile up, the context window becomes a scribbled-over scratchpad, and the model simply "shuts down."

5. Hallucinated Extra Steps

The most ironic failure: the model executes the steps and invents additional steps that don't exist—an over-eager chef adding chili sauce to a soufflé. Because LLMs are probability-driven text continuers, when text "looks unfinished," they instinctively keep writing. That creative impulse is a virtue in poetry and a fatal flaw in procedural execution.

Why "Smart" Doesn't Mean "Obedient"

These findings point to a fundamental trap: we've been using the wrong yardstick for AI. When a model passes the bar or writes elegant code, we cheer "it can reason!" But this study coldly shows: final-answer accuracy masks systematic collapse in the execution process. A model may reach a correct answer through statistical shortcuts without ever truly following the steps.

From first principles: humans executing a multi-step program maintain a mental model—which step we're on, intermediate results, what comes next—supported by working memory, metacognitive monitoring, and focus. LLMs have none of this. They process one token at a time, and "attention" is just a fuzzy guess over context. The model isn't *executing* a program; it's *guessing the most plausible continuation*—like using a weather forecast to navigate turn by turn.

A Red Light for the Agent Era

The timing matters. Tech companies are racing to ship AI agents that book flights, write code, and manage projects—tasks that all require long, multi-step, state-dependent execution. If the underlying model is only 20% reliable at 95 steps, how can we trust it with a real-world task involving hundreds of steps, like planning a wedding?

The authors offer no simple fix—because this isn't a bug but an architectural, structural defect. Real solutions may require paradigms beyond the Transformer, or at least an explicit procedural execution layer (e.g., tighter coupling with symbolic reasoning engines or deterministic execution environments).

What Would Feynman Say?

Feynman liked to say: "If you think you understand something, try to teach it to a computer." Under his standard, today's LLMs are excellent imitators of understanding and poor practitioners of execution—they produce brilliant essays on quantum mechanics but get lost at step five when asked to follow fixed procedures.

This doesn't negate the revolutionary value of LLMs—indeed, the researchers used LLMs themselves to generate test programs and evaluation data. But as Feynman said while investigating the Challenger disaster: "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."

For AI agents taking flight today, we need the same honesty: before handing critical tasks to AI, let's confirm it can actually count to ninety-five without losing its way.

Paper Details

• Title: When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
• Authors: Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh
• Institution: Indian Institute of Technology Gandhinagar (IIT Gandhinagar), et al.
• arXiv: 2605.00817
• Published: May 1, 2026
• Field: Computation and Language (cs.CL)
• Length: 77 pages, 109 figures

Key data:

• Models tested: 14 mainstream LLMs
• Datasets: 55
• Program length: 5–95 steps
• Accuracy: 61% (5 steps) → 20% (95 steps)
• Five failure modes: Missing Answers, Premature Answers, Self-Correction After Initial Error, Under-Executed Traces, Hallucinated Extra Steps

Core conclusion: High final-answer accuracy on reasoning benchmarks may conceal substantial weaknesses in faithfully executing specified program steps. Execution reliability degrades sharply as program length grows and intermediate-variable recall dependencies are introduced.

*This post is based on an arXiv preprint, written in a narrative style balancing scientific accuracy with readability.*