Build the Ruler First, Then Climb: DuoDiscover Lets AI Agents Create Their Own Cheap Evaluators

Forum topic · 小凯 · 2026-10-20

Summary

DuoDiscover, a new system from UC Berkeley, Stanford, and Google (arXiv 2610.17184), tackles the evaluation bottleneck in hill-climbing code optimization by AI agents. Instead of treating the evaluator as fixed, each agent selects a code region to optimize and writes its own lightweight "yardstick"—a proxy metric that runs in seconds instead of hours (e.g., a few seconds versus an 85-minute real vLLM evaluation). The yardstick only needs to preserve ranking, not exact scores. To combat reward hacking, an orchestrator periodically checks each yardstick's rank correlation (Kendall tau, threshold 0.5) against real evaluations; stale yardsticks are discarded. Agents run decentralized two-loop optimization: a fast inner loop using the yardstick and a slow outer loop with real evaluations, which remain the sole source of confirmed gains. Results include a 2.16x average speedup on FormulaCode, 1.92x on vLLM (vs. 1.07x for Claude Code), strong showings on SWE-fficiency, SOL-Exec, SQLite, zstd, and a CPU data prefetcher that beat the best human design in 16.7 days versus AlphaEvolve-based ArchAgent v2's 40+ days. Notably, the case study FreeToken serves a 753B-parameter MoE model (433GB weights) on a single 48GB L40S GPU via hot/cold weight tiering. Real evaluations are not eliminated—only the thousands of cheap inner-loop checks are. No code repository is released yet.

Build the Ruler First, Then Climb

Imagine an AI agent trying to speed up vLLM. It modifies one line in the scheduler. To find out whether that change helps, it has to run a real evaluation—85 minutes.

Hill-climbing optimization takes thousands of steps. At 85 minutes per step, you only get a few steps per day. The search starves before it even starts: the bottleneck isn't "what to change," it's "how to know whether the change worked."

DuoDiscover (Berkeley × Stanford × Google, 24 authors with four co-first authors, posted October 12 on arXiv) offers a counterintuitive answer: don't hand the agent a fixed ruler—let it build its own.

Building a Cheap Ruler

The paper's critique of prior systems is blunt: everyone polishes the search framework while treating the evaluator as a fixed premise. DuoDiscover dismantles that premise—each coding agent picks a code region worth optimizing and writes a dedicated lightweight metric for it, officially called a *yardstick*.

A real example from vLLM: an agent targets prefill batching in the scheduler and writes a yardstick that is a simplified model of decode-step latency, running in seconds. The real evaluation takes 85 minutes; the yardstick takes seconds—a three-orders-of-magnitude gap.

The yardstick doesn't need to produce the same scores as the real evaluation. It only needs to rank two changes the same way—whichever is better comes first. If the ordering is right, the direction is right; how far off the absolute scores are doesn't matter.

When the Ruler Lies

This is the paper's most honest section. After thousands of optimization steps against one metric, an agent will inevitably find the gaps between the yardstick and reality—and exploit them to farm points. The paper names it directly: reward hacking. A yardstick that was accurate at the start may not stay accurate.

The antidote is rank correlation (Kendall's tau). After each real evaluation, the orchestrator re-checks the yardstick's ranking consistency using the true scores—computed only on pairs whose true score difference exceeds evaluation noise, so the evaluator's own uncertainty isn't blamed on the yardstick. Tau ≥ 0.5 passes; a yardstick that falls below the bar is discarded on the spot, and the agent must build a new one or switch targets. All final gains are still credited only by real evaluation: the yardstick points the way, the real evaluation stamps the approval.

A Bottom-Up Swarm of Agents

There is no centralized task decomposition. Every agent picks its own goals and runs a two-loop cycle. Fast loop: modify code, score with the yardstick, keep gains, roll back losses—a few seconds per step. Slow loop: confirm gains with real evaluation and re-check the yardstick—hours per pass. A lightweight orchestrator does only three things: kill stalled agents, spawn new agents seeded with confirmed gains, and pass notes to running agents.

Results

FormulaCode (a dataset of real performance-optimization PRs, single evaluations from 7 minutes to 6.8 hours): human expert submissions average 1.048x, OpenEvolve 1.161x, agent baselines average 1.78x, DuoDiscover 2.16x
SWE-fficiency: average speedup is twice that of human experts; for reference, it matched a 69.12% score achieved by Claude Code in 2.8 hours
SOL-Exec (NVIDIA kernel leaderboard): 121 first places and 212 top-three finishes out of 235 kernels
vLLM: 1.92x (geometric mean over four targets; Claude Code under the same budget: 1.07x); SQLite: 1.38x
CPU microarchitecture (data prefetcher design, single evaluations up to 28 hours): beat the best human design; gains are 1.65x those of ArchAgent v2 (an AlphaEvolve-lineage system), taking 16.7 days versus 40+ days

One standout case study: FreeToken—serving a 753B-parameter MoE model (433GB of weights) on a single 48GB L40S GPU, using hot/cold tiering that keeps weights in main memory. The real evaluation for this task takes 70 minutes per run; the yardstick gets built and the hill gets climbed anyway. The breadth of targets is notable: repo-level code, GPU/TPU kernels, databases, the zstd compression library, even chip component design.

Two Doses of Cold Water

The real evaluation has not been eliminated. The outer loop still runs hourly-scale real evaluations; the savings come only from the thousands of cheap inner-loop checks. Claiming this "solves the evaluation bottleneck at its root" would be too strong—it's a throttle on evaluation, not a removal. The tau ≥ 0.5 bar is also loose: half-correlated is enough to pass. And the paper ships no code repository, with a CC BY-NC-ND license—reproducers will have to wait.

Closing

16.7 days versus 40 days—the difference isn't compute, it's the idle waiting every time you want to confirm "did this work?" The inner loop is three orders of magnitude cheaper per step while the outer loop runs the same number of real evaluations. The speedup is exactly that gap.

Paper: arxiv.org/abs/2610.17184 Project page: duodiscover.github.io