FDE-Gym: The Client Just Wanted a Summary Email

This is the final installment of a three-part "field test" series benchmarking frontier AI agents: a collaboration exam (four Gemini 4 Argon agents coordinating in Minecraft — passed, the series' only win), a timing exam (Realtime-Bench, where the best of 12 frontier models hit only 44% on real-time interaction), and now the delivery exam. FDE-Gym has no salvageable headline number: the best of 13 model configurations averaged 0.56; of 95 end-to-end tasks, only 14 were fully resolved; 5 of 13 configurations solved zero tasks.

The deeper significance is the exam design itself: it is reportedly the first benchmark that refuses to assume the task is already well-defined.

Who Is Being Tested: The Forward Deployed Engineer

An FDE (Forward Deployed Engineer) is the engineer sent to the client's site — like a contractor told only "I want my home brighter," who must discover load-bearing walls, gas lines, building rules, and budget limits on their own. The benchmark tests the full skill set of this role: clarifying vague requirements, operating on live systems, and surfacing undocumented constraints. The author team spans DeepLethe, Tsinghua University, Shanghai Jiao Tong University, Beihang University, Huawei, Meta, Microsoft Research Asia, and others — 23 core authors and contributors, with all tasks adapted by industry experts from real delivery projects.

How the Exam Is Built

• 95 end-to-end delivery tasks from 20 client scenarios across 18 industries; 130+ simulated stakeholders; 13 model configurations from 7 providers (including Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra, GPT-6.1 Sol, GLM-5.3 Flash, DeepSeek V4.1 Flash, Gemini 3.8 Flash, Qwen 3.8 Flash, GPT-5.5, GPT-6 Luna, Mistral Small 4, Claude Haiku 4.5); 10,000+ hours of expert construction.
• Tools limited to 5: run command, read/write/edit/list files. No browser, no prebuilt software.
• Each task runs 3 times with hard caps: 200 turns, 150 minutes, $8 in model usage — blocking trial-and-error, stalling, and brute-force stakeholder interrogation.

Scoring defines "delivery" itself:

• Normalized business uplift: U = (post-delivery KPI − do-nothing KPI) / (expert-reference KPI − do-nothing KPI). U = 0 means no impact; U = 1 matches the expert; U < 0 means the agent made things worse.
• Incident adjustment: any operational incident (e.g., unauthorized actions) zeroes out positive uplift while keeping negatives.
• Stakeholder contact cost C: every message, trial request, and waiting day is billed at hidden per-person rates that rise past patience limits — and agents are never told communication costs money. Final score S = adjusted U − C. "Resolved" requires S ≥ 0.8 with zero incidents.
• Anti-gaming anchors: expert reference must score ≥ 0.95; do-nothing baseline = 0; literal compliance caps at 0.2; shortcut strategies (overfitting historical data, guessing integer thresholds, deferring every uncertain decision to humans) cap at 0.8; bypassing client controls or manipulating graders caps near 0. Over 270 shortcut/bypass strategies were tested and blocked. Scenarios have 2–7 variants, and no problem pattern exceeds one-third of the task set.

All 130+ stakeholders are scripted (model swaps shift success rates by several percentage points), with fixed knowledge, misconceptions, disclosure conditions, limited patience, and availability constraints. No org chart is provided — agents must infer the org from client documents.

Four Levels of Ambiguity

Same workspace, stakeholders, and grader; only the instruction changes:

• L0: full specifications — tests only implementation.
• L1: problem stated, rules and constraints withheld.
• L2: the problem description itself becomes vague.
• L3: only the sponsor's raw request — which may not be the real problem.

Key Findings

1. The best students are far from the delivery bar

Under L3, Claude Opus 5.5 led at 0.56 (95% CI 0.50–0.62), solving 14 tasks. An oracle that picks the best configuration per task in advance averages only 0.70 (30 solved). Only one scenario (bank_scam_hold) saw any model average ≥ 0.8.

2. The cliff appears when constraints go hidden

At L0, three leading models scored 0.73–0.82. Moving to L1 dropped them to 0.39–0.62 — an average fall of 0.28 (95% CI 0.17–0.39). From L1 to L3, scores fell only another 0.04: once forced to dig for unstated constraints, strong models also discover the true need, since real constraints are visible only next to the client's actual pain. The bottleneck is constraint mining, not linguistic ambiguity. Weak models collapse at every level: GPT-5.5 falls 0.51 → 0.15 → 0.04 → 0.06; Claude Haiku 4.5 falls 0.11 → −0.11 → −0.30 → −0.25.

3. Asking the right people is not delivering

Under L3, leading models retrieve 72–85% of key facts but achieve only 0.60–0.70 normalized uplift. Stakeholder contact costs deduct a further 0.14–0.22. Claude Opus 5.5 gathered all key facts on 23 tasks but resolved only 10 at ≥ 0.8 (40 → 14 overall); GPT-6 Astra hit 0.8 uplift on 36 tasks but resolved only 9 after costs. Most models pay this bill unknowingly — the benchmark turns an unspoken workplace norm into an explicit penalty.

4. Weak models are negative assets

Claude Haiku 4.5 left clients worse than doing nothing on 79 of 95 tasks; Mistral Small 4 on 51 — before communication costs. Negative uplift is preserved in the score.

5. No universal winner

Five of 20 scenarios saw no model average ≥ 0.5; leading models' cross-task median range spans 0.34–0.44. Model selection cannot fix delivery.

The Catch-22 on the Drilling Platform

One scenario: a drilling-company VP wants a daily summary email for the board and an "Ask your wells" chat interface. The downtime-coding system is explicitly "out of scope." Literal compliance scores at most 0.2. But the real bottleneck is a keyword-based downtime script that misattributes downtime to contractors — each disputed line suspends monthly compensation. Fixing it requires touching the excluded system, identifying the applicable contract rule, confirming interpretation with the cost engineer, and waiting for an approver who is out this week. Every actor is rational; together they form a catch-all net where doing the job right means crossing a line the client drew — and crossing lines constitutes an incident in most workplaces. The sponsor wants a summary email; the company actually hurts from monthly compensation pauses. What he asks for is not what he needs.

The Map from Three Exams

• Collaboration: AI wins in the cleanest exam — explicit rules, clear goals, infinitely resettable environments.
• Real-time: AI falters when the world flows live and errors have no undo — 44% best success rate.
• Delivery: when the problem itself is undefined, the best model completes less than 60% of the path from bare walls to sign-off.

Each exam is lost differently: coordination, timing, and — in delivery — three failure modes (missed constraints, unaffordable communication, actively broken systems) that no amount of single-point capability fixes. These are structural capability gaps, not intelligence deficits.

The paper's conclusion is blunt: a large gap exists between "solving well-defined tasks" and "delivering effective AI systems for real clients." Closing it requires agents that can surface unstated constraints, operate reliably on live systems, and produce measurable improvements in client metrics — none of which current models truly pass. This is diagnosis, not despair: the platform at fde-gym.com lets humans practice FDE skills under the identical grader, with a human expert baseline being collected.

Paper Information

• Title: FDE-Gym: The Last Mile of AI Delivery
• arXiv: 2610.22350 (v1, 2026-10-15)
• Lead author: Dongxu Yang (DeepLethe, corresponding), with Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen (Shanghai Jiao Tong University), et al.
• Institutions: DeepLethe, Tsinghua, Shanghai Jiao Tong, Beihang, Xiamen, Nanjing, Beijing Jiaotong, BUPT, Tianjin, Shanghai University of Finance and Economics, Harvard, Stanford, UCL, Cornell, Huawei, Atlassian, Meta, Microsoft Research Asia, StellarCube.ai, UWits Technology, PetroChina, Kunlun Digital Technology, and others
• Open source: github.com/deeplethe/fde-gym (v1.1.0); 20 original tasks available on request, 3 public examples
• Platform: fde-gym.com