Do LLM Agents Execute the Plans They Declare? A Feynman-Style Explainer
*English adaptation of a Chinese forum post (zhichai.net) explaining Oota et al., arXiv:2609.38108, using a renovation-crew analogy.*
Key points
- The problem: In mainstream Plan+ReAct pipelines, LLM agents write a plan, then execute step-by-step. Measured across three benchmarks, only 22%-45% of trajectories actually preserve the structure of the plan they declared. The authors call this the *plan declaration-execution gap*.
- Why it happens: Execution is done by re-deciding at every step over a rolling history. As history grows, the early plan's weight in attention decays-it becomes "a yellowed page in a stack of documents." This is a mechanism problem, not a model attitude problem.
- Why it matters: When a task fails, you cannot tell from the outside whether the plan was wrong or the execution drifted. Mixing up "bad choice" and "bad discipline" leads optimization efforts in the wrong direction (e.g., only improving the planner).
- Four planning modes the paper distinguishes:
- Predefined: full blueprint fixed upfront (like a direct bus route).
- Sequential: only the next step is decided, then reassessed (like turn-by-turn hiking navigation).
- Hierarchical: decompose into stages, then into actions (like a cross-city drive with waypoints).
- Search: no preset route; trial, backtrack, retry (like combing streets for an unnamed shop).
- The fix - Planning-as-Routing: (1) the model makes a lightweight one-time *declaration* of which mode fits the task; (2) a deterministic router dispatches the task to a pattern-specific executor that enforces that mode's structure. The plan becomes a machine-readable track, not a human-readable declaration.
- Structural preservation: 100% by construction-what is declared is what executes, replacing the 22%-45% baseline.
- ALFWorld (simulated household tasks): success rate up from 0.48 to 0.92-nearly doubled purely by making execution disciplined.
- SWE-bench Verified (real software engineering): success rate up from 0.36 to 0.44-real but more modest, because ceiling there is set by problem difficulty, not execution discipline.
- Validation spans four benchmarks and three LLMs, so the conclusions are not an artifact of one setting.
- Best mode varies by environment and model: Search wins on ALFWorld (cheap trial-and-error), Hierarchical wins on SWE-bench (expensive backtracking, layered localization). This variability is exactly why upfront mode declaration/routing is justified-if one mode always won, routing would be redundant.
- The declaration step is still made by the model itself, and models do not always choose well. Few-shot examples of task-to-mode pairings improve selection on *some* benchmark/model combinations, but not all. The paper cleanly splits the problem: execution fidelity is now an engineering problem (solved); mode selection remains a science problem (partially reliable).
- Oota et al. "Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution", arXiv:2609.38108, 2026-09-29
Results
No universal mode
Open problem
The post's framing
The author serializes agent capability as three "gates": coordination (previous post), honesty/faithfulness (this paper), and evolution (next post). The recurring methodology: *wherever a virtue is needed, ask whether a mechanism can replace it*-stand-ups for coordination, work orders (routing) for faithfulness.