Key Points
- ρ = -0.13 on execution track: In CODS-2025 (149 teams on AssetOpsBench), public-leaderboard rank vs hidden-test rank yielded Spearman ρ = -0.13 (n=13, p=0.71), statistically indistinguishable from zero. Planning track ρ = 0.69 but saturated across only 8 unique scores for 20 teams.
- Three structural defects in current leaderboards: 1. *Aggregate scores collapse orthogonal dimensions* — reasoning-on vs reasoning-off shows +31pp clarity gain and -7pp hallucination, while data retrieval and agent-sequence correctness stay flat; multi-turn latency differs 4.2× across Plan-Execute vs Supervisor-Specialist despite similar single-turn Pass@1; RAG vs Knowledge Plugin shows 50–68% accuracy at 8.9–20s vs ~90% accuracy at 114–146s with 4.5–10× token inflation. 2. *LLM-as-Judge is reflexive* — PHMForge reports Krippendorff α = 0.61 for LLM judges vs 0.74–0.82 for human-human agreement. Condition Insight (O'Donncha et al., 2026) raises Condition Agreement Rate from 0.68 to 0.91 via prompt design; ARE/Gaia2 (Froger et al., 2025) achieves 0.99 precision / 0.95 recall with DAG oracle on 450 human-labeled trajectories. 3. *In-sample means fail on OOD deployment* — Exgentic cross-benchmark rank correlation ranges 0.32–0.85 across six heterogeneous benchmarks.
- 12-layer measurement framework (T1–T12):
- Core capability layers (T1–T7): Pass@k, tool-call conformance, plan quality, capability axes, cost-efficiency Pareto, failure-mode taxonomy, reproducibility.
- Deployment layers (T8–T12): deployment infrastructure, multi-turn reuse, reasoning-mode sensitivity, knowledge augmentation, evidence grounding with judge-independent verification.
- *No existing benchmark reports more than 4–5 layers; T8–T12 are absent in nearly all current benchmarks.*
- Predictive validity replaces in-sample mean:
PV(c) = α · Ȳc − β · σYc,OOD − γ · IQR(Yc), fit on Criterion A holdouts to maximize Spearman against Criteria B and C.- Three OOD criteria: (A) stratified held-out scenarios, (B) cross-subset transfer (6×6 rank-stability matrix on AssetOpsBench's 6 subsets), (C) four adversarial perturbations (synonym paraphrasing, identifier renaming, time-window shift, distractor injection).
- Falsifiable conditions: 1. ρ between in-sample and OOD rankings < 0.85 on at least two criteria (already partially supported by ρ = -0.13). 2. Top-3 falling outside Top-5 ≥ 10%. 3. Pearson ρ between mean and OOD-variance > 0.2. 4. Jaccard between PV-ranking and mean-ranking Top-10 < 0.85.
- Reference: arXiv:2606.19704, Dhaval C. Patel et al., IBM Research. AssetOpsBench has 1600+ GitHub stars, 230+ forks, and EMNLP/NeurIPS/AAAI publications.
Implications
The paper reframes agent benchmarking as a methodology problem rather than a model problem: deployments care about trade-offs across latency, token cost, multi-turn reuse, retrieval strategy, and reasoning-mode sensitivity, none of which survive aggregation into a single score. Predictive validity, measured under held-out, cross-subset, and adversarial shifts, is proposed as the replacement ranking criterion, with the 12-layer framework ensuring orthogonal dimensions are reported separately rather than folded.