FDE-Gym: A Benchmark for the Last Mile of AI Delivery by Forward-Deployed Engineers

Forum topic · 小凯 · 2026-10-17

Summary

FDE-Gym (arXiv: 2610.22350) is a new benchmark evaluating AI agents on the full cycle of real-world customer delivery, the work typically done by forward-deployed engineers (FDEs). Unlike existing agent benchmarks built on well-specified tasks from issue reports or tickets, FDE-Gym contains 95 end-to-end delivery tasks adapted by industry experts from 20 customer scenarios spanning 18 sectors. Agents operate in simulated customer environments featuring scripted stakeholders and modifiable production-like systems: they must clarify ambiguous requirements, interact with stakeholders, and deliver working solutions. Scoring measures how much a delivery improves the customer's own success metric (normalized between 0 and 1) while penalizing excessive stakeholder interactions. Results show a substantial gap between benchmark-style task solving and real delivery: the best configuration achieved an average score of 0.56 and fully solved only 14 of 95 tasks, with uncovering unstated constraints identified as the primary bottleneck. The findings quantify the gap between well-defined task completion and delivering effective AI systems to real customers.

Paper Overview

Field: NLP
Authors: Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang
Published: 2026-10-15
arXiv: 2610.22350

Abstract (translated)

Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. FDE-Gym is a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged) and full improvement, while penalizing excessive stakeholder interactions.

Key Findings

The best agent configuration reached an average score of only 0.56 in realistic scenarios requiring discovery of unstated constraints.
Only 14 out of 95 tasks were fully solved.
Uncovering unstated constraints is the primary bottleneck for agents in real-world delivery.
The results reveal a significant gap between well-specified task solving and delivering effective AI systems to real customers.