PACE: A Proxy for Agentic Capability Evaluation

> Paper: *PACE: A Proxy for Agentic Capability Evaluation* > Authors: Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig > Institutions: Carnegie Mellon University / Salesforce AI Research > Paper link: https://arxiv.org/abs/2607.02032 > Code: https://github.com/neulab/pace-bench > One-liner: Agent evaluation is too expensive and slow? Use 100 atomic-capability questions to predict full benchmark results with near-full accuracy at less than 1% of the cost.

---

1. The Pain of Agent Evaluation: Expensive, Slow, Hard

Anyone doing agent research shares the same nightmare: running a full evaluation.

Take SWE-Bench as an example:

• Each model must handle real GitHub issues, call tools, read code, write patches, run tests.
• Per-model evaluation cost: thousands of dollars
• Per-model evaluation time: days
• Requires complex infrastructure: Docker, GPUs, API keys, parallel scheduling

GAIA, WebArena, and AppArena are similar. Their common trait is being end-to-end, real-environment, multi-step interactive, execution-verified.

Consequences:

• Small teams cannot afford it — no compute, no budget, no infra team.
• Model development cycles are bottlenecked by evaluation — changes take days to assess.
• Model selection becomes guesswork — you cannot afford the benchmark that would tell you which model fits your scenario.

By contrast, non-agent benchmarks (MMLU, HumanEval, GSM8K, etc.) are like unit tests:

• Fast (seconds)
• Cheap (near-zero cost)
• Simple (no complex environment)

PACE's core question: Can cheap unit tests predict expensive integration tests?

---

2. Core Idea: From Atomic Capabilities to System Capabilities

PACE's intuition is straightforward:

> Agent capability is a composition of atomic capabilities. If you know how a model performs on reasoning, code, math, tool use, and other atomic tasks, you can theoretically predict its performance on agent tasks that compose those abilities.

The real challenges are not whether prediction is possible, but which atomic items to pick, how to weight them, and whether the prediction is accurate enough.

The PACE framework has three steps.

Step 1: Build the Candidate Pool

Collect a large pool of atomic-capability instances from 19 non-agent benchmarks, covering:

• Reasoning (MMLU, BIG-Bench-Hard, etc.)
• Code generation (HumanEval, MBPP, etc.)
• Mathematics (GSM8K, MATH, etc.)
• Tool use (various API-calling benchmarks)
• Long context (L-Eval, RULER, etc.)
• Instruction following (IFEval, etc.)

Step 2: Dual-Criterion Instance Selection (Core Innovation)

Select a small set (~100 instances) with the highest predictive power. PACE uses two complementary signals:

Local signal (Target-Relevance):

• Measures rank correlation between a single instance and the target agent benchmark.
• If model A beats model B on this item, does model A also beat model B on the target benchmark?
• Picks items where local rankings reflect global rankings.

Global signal (Globally Informative):

• Uses SVD leverage scores to pick high-information instances.
• Avoids redundant items that overlap heavily.
• Ensures diversity across capability dimensions.

Combined: items must be both relevant and diverse.

Step 3: Noise-Stable Regression

Use scores from the selected ~100 instances to regress and predict the target benchmark score.

A key issue: target labels (agent benchmark scores) are themselves noisy — running the same model multiple times can yield different results. PACE uses bootstrap-resampled least-squares regression to reduce label noise and stabilize predictions.

---

3. Empirical Results: A Strong Accuracy–Cost Tradeoff

Generalization Across Models and Benchmarks

Tested on 14 models and 4 agent benchmarks (SWE-Bench, GAIA, WebArena, AppArena) with 19 non-agent benchmarks:

| Metric | Result | |---|---| | LOOCV MAE (leave-one-out cross-validation mean absolute error) | < 4% | | Spearman correlation | > 0.80 | | Pairwise model ranking accuracy | ~85% | | Cost | < 1% of full agent evaluation cost |

In practical terms:

• If GPT-5 and Claude 4 score 85 and 82 on PACE, they will very likely maintain that order on SWE-Bench.
• 100 atomic items (a few minutes, a few dollars) can predict full-benchmark outcomes (days, thousands of dollars).

Generalization to Unseen Models

The most important question: can instances selected on a training set of known models predict new models not seen during selection?

Answer: Yes. Across four major agent benchmarks, Spearman correlation stays above 0.8 and ranking accuracy stays around 85%.

This shows PACE selects items that capture true atomic-capability composition rather than overfitting to specific models.

Accuracy–Cost Tradeoff

Prediction accuracy rises smoothly with proxy size:

• 50 instances: already usable.
• 100 instances: the sweet spot (near saturation).
• Hundreds of instances: diminishing returns.

Researchers can flexibly choose scale based on their compute budget.

---

4. Interpretable Capability Analysis: What Does Each Agent Task Need?

PACE also enables analysis of which atomic capabilities drive each agent benchmark.

By inspecting which non-agent benchmarks the selected proxy items come from, you can infer:

• SWE-Bench depends most on: code generation + long context + tool use.
• GAIA depends most on: reasoning + multimodality + knowledge retrieval.
• WebArena depends most on: web understanding + navigation + form interaction.
• AppArena depends most on: GUI understanding + action sequencing + state tracking.

This guides model optimization:

• To improve on SWE-Bench, prioritize code and long-context capability, rather than blindly boosting general reasoning.

---

5. Why This Matters

1. Democratizing Evaluation

Agent evaluation has long been a game for well-resourced labs. PACE lets small teams run legitimate agent-capability assessment:

• 100 atomic items, done in minutes.
• No Docker, no GPU cluster, no complex infra.
• Accurate enough for model selection, iteration, and capability analysis.

2. A Revolution in Development Velocity

The core model development loop — train, evaluate, analyze, improve — is compressed from days to minutes:

• Faster iteration: see rough impact immediately after a model change.
• Early elimination: skip full evaluation on weak models.
• Capability diagnosis: identify and target specific weaknesses.

3. Scientific Basis for Model Selection

Enterprises deploying agents face real choices: GPT-5, Claude 4, or an open-source model? PACE offers low-cost prediction:

• Run 100 atomic items, estimate each model's target-scenario performance.
• Avoid blind "buy the most expensive" or "follow the crowd" decisions.

4. From Black-Box Scoring to Interpretable Analysis

Traditional agent evaluation gives you a score but not a reason. PACE predicts the score *and* reveals capability structure:

• Which atomic capabilities does this agent task require?
• How does my model perform on those atomic capabilities?
• Which atomic capability, if improved, would most lift the target score?

That is an upgrade from *scoring* to *diagnosis*.

---

6. Limitations and Boundaries

1. Proxy ≠ real evaluation. PACE predicts; it does not replace full evaluation. For critical decisions (paper publication, product launch), full agent evaluation remains the final arbiter. PACE's value lies in the development and screening phases. 2. Atomic-capability assumption. PACE assumes agent capability decomposes into a combination of atomic capabilities. For emergent behaviors or tasks tightly coupled to specific toolchains, this assumption may not fully hold. 3. Target benchmark coverage. The paper evaluates four agent benchmarks. PACE's effectiveness on more diverse agent tasks (robotics, scientific experimentation, creative writing) needs further verification. 4. Regression-model limits. Linear regression assumes a linear relationship between atomic and agent capability. Reality may be nonlinear with interactions. Bootstrap resampling mitigates part of the label-noise issue, but the model form itself is a simplification. 5. Non-agent benchmark coverage. PACE depends on existing non-agent benchmark pools. If a critical atomic capability is not covered by existing benchmarks, PACE cannot capture it.

---

7. One-Sentence Takeaway

PACE's core insight: Agent capability is not magic — it is a composition of atomic capabilities. Choose the right atomic items and the right prediction method, and 100 unit tests can predict the result of a full integration test.

This is not just an evaluation cost optimization; it is a methodological upgrade for agent research — from black-box scoring to interpretable capability diagnosis. Anyone doing agent development, model selection, or capability analysis should study PACE and integrate it into their workflow.

---

Reference

• Paper: *PACE: A Proxy for Agentic Capability Evaluation* (arXiv:2607.02032)
• Authors: Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig
• Institutions: CMU / Salesforce AI Research
• Code: github.com/neulab/pace-bench
• Models tested: 14 LLMs (closed and open source)
• Target agent benchmarks: SWE-Bench, GAIA, WebArena, AppArena
• Source non-agent benchmarks: 19 (MMLU, HumanEval, GSM8K, MATH, etc.)
• Key metrics: MAE < 4%, Spearman > 0.80, ranking accuracy ~85%, cost < 1%
• Techniques: Dual-criterion instance selection (local relevance + global informativeness) + bootstrap least-squares regression