DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows

Forum topic · 小凯 · 2026-05-21

Summary

DecisionBench is a new benchmark substrate for studying emergent delegation in long-horizon agentic workflows, introduced by Yuxuan Gao, Megan Wang, and Yi Ling Yu (arXiv 2505.01259). The substrate fixes a task suite (GAIA, tau-bench, BFCL multi-turn), a peer-model pool of 11 models from 7 vendor families, a delegation interface (call_model plus an optional read_profile channel), a deterministic skill-annotation layer, and multi-axis metrics covering quality, cost, latency, delegation rate, routing fidelity-at-k, vendor self-preference, and a counterfactual-delegation ceiling. Because it is agnostic to how peer information is generated or delivered, learned routers, peer memories, adaptive profile construction, and multi-step delegation can all be evaluated on it. A five-condition reference sweep over 23,375 task instances shows that mean end-task quality is statistically indistinguishable across awareness conditions (|beta| <= 0.010, p >= 0.21), routing fidelity-at-1 ranges from 7.5% to 29.5% at near-equal quality, and perfect delegation would add 15-31 percentage points over measured performance. The substrate, annotation layer, analysis pipeline, and 220 per-condition run archives are released publicly.

Paper Overview

Research areas: cs.AI, cs.CL, cs.MA Authors: Yuxuan Gao, Megan Wang, Yi Ling Yu Published: 2026-05-21 arXiv: 2505.01259

Introduction

We introduce DecisionBench, a benchmark substrate for emergent delegation in long-horizon agentic workflows. The substrate fixes:

A task suite (GAIA, tau-bench, BFCL multi-turn)
A peer-model pool (11 models, 7 vendor families)
A delegation interface (call_model plus an optional read_profile channel)
A deterministic skill-annotation layer
A multi-axis metric suite covering quality, cost, latency, delegation rate, routing fidelity-at-k, vendor self-preference, and a counterfactual-delegation ceiling

The substrate is agnostic to how peer information is generated or delivered, so learned routers, richer peer memories, adaptive profile construction, and multi-step delegation can all be evaluated against it.

Key Findings

The substrate was characterized with a five-condition reference sweep on the full pool (n=23,375 task instances). Three benchmark-level findings emerge:

1. Quality-only evaluation misses the orchestration signal: mean end-task quality is statistically indistinguishable across the four awareness conditions (|beta| <= 0.010, p >= 0.21). 2. Routing fidelity varies widely at near-equal quality: routing fidelity-at-1 ranges from 7.5% to 29.5% across conditions, with the delivery channel (on-demand tool vs. preloaded description) dominating description content. 3. Large unrealized headroom: a counterfactual ceiling places perfect delegation 15-31 percentage points above measured performance on every suite, locating substantial room for future orchestration methods.

Release

The authors release the substrate, annotation layer, reference intervention suite, analysis pipeline, and 220 per-condition run archives.

Original Abstract

We introduce DecisionBench, a benchmark substrate for emergent delegation in long-horizon agentic workflows. The substrate fixes a task suite (GAIA, tau-bench, BFCL multi-turn), a peer-model pool (11 models, 7 vendor families), a delegation interface (call_model plus an optional read_profile channel), a deterministic skill-annotation layer, and a multi-axis metric suite covering quality, cost, latency, delegation rate, routing fidelity-at-k, vendor self-preference, and a counterfactual-delegation ceiling. The substrate is agnostic to how peer information is generated or delivered, so learned routers, richer peer memories, adaptive profile construction, and multi-step delegation can all be evaluated against it. We characterize the substrate with a five-condition reference sweep on the full pool (n=23,375 task instances). Three benchmark-level findings emerge: (i) mean end-task quality is statistically indistinguishable across the four awareness conditions (|beta| <= 0.010, p >= 0.21), so quality-only evaluation would miss the orchestration signal; (ii) routing fidelity-at-1 ranges from 7.5% to 29.5% across conditions at near-equal mean quality, with delivery channel (on-demand tool vs. preloaded description) dominating description content; (iii) a counterfactual ceiling places perfect delegation 15-31 percentage points above measured performance on every suite, locating large unrealized headroom for future orchestration methods. We release the substrate, annotation layer, reference intervention suite, analysis pipeline, and 220 per-condition run archives.

--- *Auto-collected on 2026-05-21*