[论文] PLE: Physical-Engineering's Last Exam

论文概要

研究领域: ML 作者: Rujun Han, Yufan Zhuang, Arijit Ray, Han Yu, Zoey CuiZhu, George Lee et al. 发布时间: 2026-10-24 arXiv: 2610.30537

中文摘要

前沿语言模型在流行知识基准上已超 90 分，越来越多以智能体形式承担数小时的专业工作。但现有基准无一检验它们能否完成执业物理工程师真实、有偿的工作：跨多格式项目语料建模物理与法规要求、优化材料与安装成本、签发持证审核员会签字的 2D/3D 工程设计包。现有工程基准集中在教科书问答、孤立工具调用或自包含的玩具设计题——多数已被领先智能体饱和，且离交付真实工程设计很远。本文推出 PLE：85 个长程智能体任务组成的专业物理工程基准，覆盖电气、土木、结构与机械工程；完成一个任务意味着交付一个真实可计费的工程成果。PLE 结合两条互补的策划路径：专家撰写任务由专业数据供应商的执业工程师端到端创建；共创任务从执业工程师写的问题陈述出发，由对抗式 Creator–Solver 智能体流水线构建每个任务的真实项目环境、可执行评分器与参考交付物，再交回执业工程师验证并加大难度。在七种公开智能体 harness 下评估多种前沿 LLM：最强配置平均仅 50.6 分，且平均每任务消耗 2700 万 token、63 分钟——当前智能体与自主物理工程设计之间存在实质鸿沟。

原文摘要

Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work. Yet no existing benchmark tests whether they can perform the real-world, economically valuable work of a professional physical engineer, such as modelling physics and legal code requirements across a multi-format project corpus, optimising material and installation costs, and issuing 2D/3D engineering design packages that a licensed reviewer would sign off on. Existing engineering benchmarks focus on textbook question answering, isolated tool calls or self-contained toy design problems. As a result, most of these benchmarks are already saturated by leading agents and solving them is far away from delivering real engineering design. We introduce PLE, a benchmark of 85 long-horizon, agentic tasks in professional physical engineering consisting of electrical, civil, structural and mechanical engineering, where solving a task means completing a realistic, billable engineering deliverable. PLE combines two complementary curation paths. Expert-Authored tasks are created end to end by practising engineers from a professional data provider's expert pool. Co-Authored tasks start from problem statements written by practising engineers; an adversarial Creator-Solver agent pipeline constructs each task's realistic project environment, executable grader and reference deliverables before sending them to expert engineers for validation and difficulty intensification. We evaluate a variety of frontier LLMs under seven publicly available agent harnesses. Even the strongest configuration reaches a mean score of only 50.6%, while spending 27M tokens and 63 minutes per task on average, demonstrating the substantial gap between current agents and autonomous physical engineering design.

---

*自动采集于 2026-10-24*

#论文 #arXiv #ML #小凯