An IBM position paper argues that current LLM agent leaderboards fail to predict real-world deployment performance.
Key points
- ρ = -0.13 between public and hidden rankings. Data from the CODS-2025 competition (149 teams, based on AssetOpsBench) shows the execution track's public-hidden Spearman rank correlation was ρ = -0.13 (n=13, p=0.71)—statistically indistinguishable from zero. The planning track showed ρ = 0.69 (n=20), but suffered from score saturation: only 8 unique scores across 20 teams.
- Three structural flaws of existing leaderboards: 1. *Aggregate scores fold orthogonal dimensions.* Configurations with identical total scores can have opposite trade-offs (reasoning-heavy vs retrieval-rich vs tool-fragile). Examples: reasoning-on vs off differ by 31pp in clarity (61% → 92%) and 7pp in hallucination rate while leaving retrieval accuracy unchanged; Plan-Execute vs Supervisor-Specialist architectures score similarly on single-turn Pass@1 but differ 4.2× in round 2-5 latency due to cross-turn artifact reuse; single-pass RAG (50-68% accuracy, 8.9-20s) vs multi-hop Knowledge Plugin (~90% accuracy, 114-146s, 4.5-10× token inflation) have no single winner. 2. *LLM-as-Judge is reflexive.* On PHMForge, judge inter-rater reliability was Krippendorff α = 0.61, below human-human α of 0.74-0.82 on the same data. Judge-independent alternatives work: Condition Insight raised Condition Agreement Rate from 0.68 to 0.91 via prompt design alone; ARE/Gaia2's DAG oracle achieved 0.99 precision / 0.95 recall on 450 human-annotated trajectories. 3. *Out-of-distribution behavior is the deployment problem.* Held-out, cross-domain, and adversarially perturbed scenarios are unpredictable from in-sample averages. Cross-benchmark analysis found rank correlations of only 0.32-0.85 across six heterogeneous benchmarks.
- A 12-layer measurement framework consolidating 7 existing benchmarks and 14 industrial implementation studies. T1-T7 cover core capabilities (pass-rate floors, tool-calling correctness, plan quality, capability axes, cost-efficiency Pareto, failure taxonomy, reproducibility). T8-T12 cover deployment extensions (infrastructure, multi-turn dialogue, reasoning-mode sensitivity, knowledge augmentation, evidence grounding with judge-independent verification). No existing benchmark reports more than 4-5 layers; T8-T12 are almost entirely missing.
- Predictive validity replaces mean scores. The paper proposes ranking by the Spearman correlation between in-sample and out-of-sample ranks, tested under three OOD criteria: (A) held-out scenarios, (B) cross-subset transfer (yielding a 6×6 rank-stability matrix on AssetOpsBench), and (C) adversarial perturbation (paraphrase, identifier renaming, time-window shift, distractor injection). A formal PV Score combines mean score, OOD rank variance, and per-scenario IQR; weights are left for future empirical work.
- Four falsifiable conditions, including in-sample vs OOD rank ρ ≥ 0.85 (if generalization were fine), top-3 fall-out rate < 10%, mean-vs-OOD-variance Pearson ρ > 0.2, and PV-ranking vs mean-ranking Top-10 Jaccard ≥ 0.85. The first condition is already partially supported by the execution-track ρ = -0.13.
- PHMForge ablations (99 SME-authored scenarios, 39 algorithm-grounded MCP tools) show orchestration errors dominate: replacing MCP tool execution with text-RAG dropped a pass-all-3 metric from 100% to 20%; cross-device transfer fell from 84.1% to 42.7%; frontier LLMs are "stronger at calling tools than at planning when to call them."
- Paper: https://arxiv.org/abs/2606.19704
- Authors: Dhaval C. Patel et al., IBM Research; contact: pateldha@us.ibm.com
- Benchmarks cited: AssetOpsBench, PHMForge, ARE/Gaia2
- Related precedent: Recht et al. 2019 ImageNet distribution-shift studies
The paper's core claim: current leaderboard total scores systematically underestimate the dimensional complexity of deployment; predictive validity plus multi-dimensional reporting should replace single-score rankings.
References