PACE: A Proxy for Agentic Capability Evaluation
> Paper: PACE: A Proxy for Agentic Capability Evaluation > Authors: Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig > Affiliations: Carnegie Mellon University / Salesforce AI Research > Paper: https://arxiv.org/abs/2607.02032 > Code: https://github.com/neulab/pace-bench > TL;DR: Agent evaluation is expensive and slow. PACE uses ~100 atomic-capability questions to predict full agent benchmark results with near full-evaluation accuracy at under 1% of the cost.
---
1. The Pain of Agent Evaluation: Expensive, Slow, Hard
Anyone doing agent research shares one nightmare: running the evaluation.
Take SWE-Bench as an example:
- Each model must handle real GitHub issues: calling tools, reading code, writing patches, running tests
- Per-model evaluation cost: thousands of dollars
- Per-model evaluation time: days
- Requires complex infrastructure: Docker, GPUs, API keys, parallel scheduling
- Small teams simply can't afford it—no compute, no budget, no infra team
- Model development cycles are dragged down by evaluation—wait days to know if a change helped
- Model selection is guesswork—you can't tell which model fits your scenario because you can't afford to evaluate
- Fast (seconds)
- Cheap (near-zero cost)
- Simple (no complex environment needed)
- Reasoning (MMLU, BIG-Bench-Hard, etc.)
- Code generation (HumanEval, MBPP, etc.)
- Math (GSM8K, MATH, etc.)
- Tool use (various API-calling benchmarks)
- Long context (L-Eval, RULER, etc.)
- Instruction following (IFEval, etc.)
- Measures each instance's rank correlation with the target agent benchmark
- If model A beats model B on this question, does A also beat B on the target benchmark?
- Picks questions whose "local ordering reflects global ordering"
- Uses SVD leverage scores to pick high-information instances
- Avoids selecting many highly correlated (redundant) questions
- Ensures coverage across capability dimensions (diversity)
- If GPT-5 and Claude 4 score 85 and 82 on PACE, their true SWE-Bench ordering will likely match
- 100 atomic questions (minutes, a few dollars) can predict full evaluation (days, thousands of dollars)
- 50 instances: already usable
- 100 instances: sweet spot (accuracy near saturation)
- Beyond a few hundred: diminishing returns
- SWE-Bench needs most: code generation + long context + tool use
- GAIA needs most: reasoning + multimodality + knowledge retrieval
- WebArena needs most: web understanding + navigation + form interaction
- AppArena needs most: GUI understanding + action sequences + state tracking
- To improve SWE-Bench scores, prioritize strengthening code and long-context abilities
- Rather than blindly scaling generic reasoning
- 100 atomic questions, done in minutes
- No Docker, no GPU clusters, no complex infra
- Accuracy sufficient for model selection, iteration evaluation, and capability analysis
- Fast iteration: know the approximate effect immediately after a change
- Early elimination: don't waste full evaluation on bad models
- Capability diagnosis: know where the weaknesses are and improve targeted
- Run 100 atomic questions to estimate each model's performance in the target scenario
- Avoid blind decisions like "buy the most expensive" or "follow the crowd"
- Which atomic capabilities does this agent task need?
- How does my model perform on each atomic capability?
- Which atomic capability improvement gives the biggest gain?
- Paper: PACE: A Proxy for Agentic Capability Evaluation (arXiv:2607.02032)
- Authors: Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig
- Affiliations: CMU / Salesforce AI Research
- Code: https://github.com/neulab/pace-bench
- Models tested: 14 LLMs (closed and open source)
- Target agent benchmarks: SWE-Bench, GAIA, WebArena, AppArena
- Source non-agent benchmarks: 19 (MMLU, HumanEval, GSM8K, MATH, etc.)
- Key metrics: MAE < 4%, Spearman > 0.80, ranking accuracy ~ 85%, cost < 1%
- Techniques: dual-criterion instance selection (local relevance + global informativeness) + bootstrap least-squares regression
GAIA, WebArena, and AppArena are the same. These agent benchmarks share defining traits: end-to-end, real environments, multi-step interaction, execution-based verification.
Consequences:
By contrast, non-agent benchmarks (MMLU, HumanEval, GSM8K, etc.) are like unit tests:
PACE's core question: Can cheap unit tests predict expensive integration tests?
---
2. PACE's Core Idea: From Atomic Capabilities to System Capability
PACE's intuition is simple:
> Agent capability is a composition of atomic capabilities. If you know how a model performs on atomic skills—reasoning, code, math, tool use—you can in principle predict its performance on agent tasks that combine them.
The key challenge: not "is prediction possible" but "which atomic questions to use, how to weight them, and is the prediction accurate enough."
The PACE framework has three steps:
Step 1: Build a Candidate Pool
Collect a large pool of atomic-capability instances from 19 non-agent benchmarks covering:
Step 2: Dual-Criterion Instance Selection (Core Innovation)
Select a small subset (~100) of the most predictive instances using two complementary signals:
Local signal (Target-Relevance):
Global signal (Globally Informative):
Combined: questions must be both "relevant" and "diverse."
Step 3: Noise-Stable Regression
Use scores on the ~100 selected instances to regressively predict scores on the target agent benchmark.
Key issue: target labels (agent benchmark scores) are noisy—running an agent benchmark multiple times on the same model can give different results. PACE uses least-squares regression with bootstrap resampling to reduce noise and improve prediction stability.
---
3. Results: An Excellent Accuracy-Cost Balance
Generalization Across Models and Benchmarks
Tested on 14 models, 4 agent benchmarks (SWE-Bench, GAIA, WebArena, AppArena), and 19 non-agent benchmarks:
| Metric | Result | |------|------| | LOOCV MAE (leave-one-out cross-validation mean absolute error) | < 4% | | Spearman correlation | > 0.80 | | Pairwise model-ranking accuracy | ~ 85% | | Cost | < 1% of full agent evaluation cost |
This means:
Generalization to Unseen Models
The most critical question: can proxy instances selected on known models predict new, unseen models?
Answer: yes. Across all 4 agent benchmarks, Spearman correlation exceeds 0.8 and ranking accuracy is about 85%.
This shows PACE's selections are not overfit to specific models—they genuinely capture the atomic capability composition of agent ability.
Cost-Accuracy Tradeoff
Prediction accuracy rises smoothly with the number of proxy instances:
Teams can freely choose scale based on compute budget.
---
4. Interpretable Capability Analysis: What Do Agent Tasks Actually Need?
Another value of PACE: analyzing which atomic capabilities each agent benchmark most depends on.
By looking at which non-agent benchmarks the selected proxy instances come from, one can infer:
This guides model optimization:
---
5. Why This Matters
1. Democratizing Evaluation
Agent evaluation has been a game for top players. PACE lets small teams do rigorous agent capability assessment:
2. A Revolution in Development Efficiency
The core development loop—train → evaluate → analyze → improve—goes from "days" to "minutes":
3. Scientific Grounding for Model Selection
Enterprises deploying agents face a selection problem: GPT-5, Claude 4, or open-source? PACE provides low-cost prediction:
4. From "Black-Box Scoring" to "Interpretable Analysis"
Traditional agent evaluation gives a score but not a reason. PACE not only predicts scores but reveals capability structure:
This is an upgrade from "scoring" to "diagnosis."
---
6. Limitations and Boundaries
1. Proxy evaluation ≠ real evaluation. PACE predicts, it does not replace. For critical decisions (paper publication, product launch), full agent evaluation is still needed as final validation. PACE's value lies in the development and screening stages.
2. The atomic-capability assumption. PACE assumes agent capability decomposes into atomic capabilities. For emergent behaviors or tasks highly dependent on specific toolchains, this may not fully hold.
3. Target benchmark coverage. The paper tested 4 agent benchmarks. For more diverse agent tasks (robot manipulation, scientific experiments, creative writing), PACE's effectiveness needs verification.
4. Regression model limitations. Linear regression assumes a linear relationship between atomic and agent capabilities; reality may be nonlinear with interaction terms. Bootstrap resampling mitigates some noise, but the model form is a simplification.
5. Dependence on the non-agent benchmark pool. PACE relies on existing non-agent benchmarks. If key atomic capabilities aren't covered by existing benchmarks, PACE can't capture them.
---
7. One-Line Summary
PACE's core insight: agent capability is not magic—it's a composition of atomic capabilities. Pick the right atomic questions and the right prediction method, and 100 unit tests can predict the outcome of full integration tests.
This is not just a cost optimization but an upgrade to agent research methodology—from "black-box benchmarking" to "interpretable capability diagnosis." For anyone doing agent development, model selection, or capability analysis, PACE deserves careful study and integration into your workflow.
---