[论文] Testing the Planning Claims of LLM-Based Agents against Formal Sema...

论文概要

研究领域: ML
作者: Oriol Miro-Lopez-Feliu, Oscar Miró López-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao
发布时间: 2026-10-19
arXiv: 2610.22469

中文摘要

LLM 智能体日益被赋予规划、监控、自我修正等实践能力，其成功常被归因于围绕模型构建的提示策略与智能体架构——思维链、搜索式提示、自我精炼、反思、批评者与工具循环。但这些主张通常由端到端成功率（只显示任务是否完成）或 LLM 评审（并非验证器）支撑。本文用形式语义检验 LLM 规划：任务以自然语言提出，但每个任务都落地于一个隐藏的 PDDL 模型，由它判定每个动作是否可执行、目标是否达成、规划是否存在。跨五个领域（两个全新发布，全部随论文公开）、四个模型、六种提示策略、三种智能体架构的结果：最佳配置离线解决 78% 可解任务，但在线只有 40%——智能体大多不检查状态就行动；不可解任务在不到 10% 的回合中被识别；更重的提示策略比直接提示多用 5 到 22 倍 token 却无明确收益；规划性能在不同模型间的差异大于在不同提示策略或架构之间。作者结论：声称智能体「会规划」，必须拿出经验证的动作轨迹作为证据。

原文摘要

LLM-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain-of-thought, search-based prompting, self-refinement, reflection, critics and tool-using loops. Yet these claims are usually supported by end-to-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers. We test LLM planning against formal semantics. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all. Across five domains (two of them new, all released with the paper), four models, six prompting strategies and three agent architectures, the best configuration solves 78% of the solvable tasks offline but only 40% online, where agents mostly act without inspecting the state. Unsolvable tasks are recognised in under 10% of episodes, and heavier prompting strategies use 5 to 22 times as many tokens as direct prompting with no clear gain. In our experiments, planning performance varied more across models than across prompting strategies or agent architectures. Claims that agents can plan require evidence from validated action traces.

自动采集于 2026-10-19

#论文 #arXiv #ML #小凯