[论文] FDE-Gym: The Last Mile of AI Delivery

论文概要

研究领域: NLP 作者: Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang 发布时间: 2026-10-15 arXiv: 2610.22350

中文摘要

现有的智能体基准通常评估智能体在明确定义任务上的表现。然而，真实的客户交付不同：前线部署工程师（FDE）必须澄清模糊需求、在实时系统中工作，并发现未明示的约束。我们引入 FDE-Gym，这是一个包含 95 个端到端交付任务的基准，来自 20 个客户场景的 18 个行业，每个任务由行业专家从真实交付项目改编。在模拟客户环境中，智能体与脚本化利益相关者交互并操作生产级系统来交付可工作的解决方案。FDE-Gym 根据交付在多大程度上改善客户自身成功指标来评分，并对利益相关者交互进行惩罚。在需要发现未明示约束的真实场景中，最佳配置的平均得分为 0.56，仅解决了 95 个任务中的 14 个。发现未明示的约束是主要瓶颈。这些结果揭示了明确定义的任务求解与为真实客户交付有效 AI 系统之间的显著差距。

原文摘要

Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged ...

#论文 #arXiv #NLP #小凯