Testing LLM Agent Planning Claims Against Formal PDDL Semantics

Forum topic · 小凯 · 2026-10-19

Summary

This arXiv paper (2610.22469) evaluates whether LLM-based agents truly perform practical reasoning—planning, monitoring, and self-correction—by testing them against formal semantics. Tasks are stated in natural language but grounded in hidden PDDL models that verify action applicability, goal satisfaction, and plan existence. Across five domains (two newly released), four models, six prompting strategies, and three agent architectures, the best configuration solves 78% of solvable tasks offline but only 40% online, where agents act mostly without inspecting the state. Unsolvable tasks are recognized in under 10% of episodes, and heavier prompting strategies (chain-of-thought, self-refinement, reflection, critics, tool loops) consume 5 to 22 times more tokens than direct prompting with no clear gain. Notably, planning performance varies more across models than across prompting strategies or architectures. The authors conclude that claims of agent planning capability require evidence from validated action traces rather than end-to-end success rates or LLM-judge evaluations.

Paper Overview

Field: Machine Learning

arXiv: 2610.22469

Authors: Oriol Miro-Lopez-Feliu, Oscar Miró López-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao

Abstract

LLM-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain-of-thought, search-based prompting, self-refinement, reflection, critics and tool-using loops. Yet these claims are usually supported by end-to-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers.

Methodology

The authors test LLM planning against formal semantics. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all.

Experimental scope:

Five domains (two of them new, all released with the paper)
Four models
Six prompting strategies
Three agent architectures

Key Findings

The best configuration solves 78% of solvable tasks offline, but only 40% online, where agents mostly act without inspecting the state.
Unsolvable tasks are recognized in under 10% of episodes.
Heavier prompting strategies use 5 to 22 times as many tokens as direct prompting, with no clear gain.
Planning performance varied more across models than across prompting strategies or agent architectures.

Conclusion

Claims that agents can plan require evidence from validated action traces—not just end-to-end success rates or LLM-judge evaluations.

---

*Auto-collected on 2026-10-19*