AI Harnesses Are Transferable Programs, Model Weights Are Not: Google's Harness² Paper Explained

Forum topic · 小凯 · 2026-10-18

Summary

A Google Cloud AI Research paper introduces Harness², a recursive framework that lets AI agents improve their own harnesses—the surrounding setup of prompts, tools, memory, and orchestration logic—without needing ground-truth answers. Instead of grading outputs, the system runs controlled A/B-style experiments: propose harness changes, run the original and modified versions on identical tasks, and only accept changes backed by consistent behavioral differences. Successful changes plus reusable lessons are stored in an experience library, subject to a strict admission rule: only execution-comparison-supported insights get saved. On WorkBuddy-Bench a single improvement step gains up to 11.5 points; LAB gains 11.8 points; removing the library causes scores to fall. The strongest evidence is cross-model transfer: a harness tuned for Gemini 3.5 Flash, applied unchanged to GLM-5, lifts code-task scores from 58.5 to 75.8, and a harness-boosted GLM-5 (75.3) beats stock Gemini (72.4). The paper frames the core insight as: harnesses are programs—copyable and transferable—while weights are muscle. Costs multiply with every candidate run, and the experience library is only a simple index so far. The framework is open source (Apache 2.0, google-research/harness2), with source arXiv 2610.14224.

AI's Gear Can Be Gifted, Muscle Cannot

Start with one number. Google researchers carefully tuned a full set of gear for the Gemini 3.5 Flash model—prompts, tools, orchestration logic—and then fitted this entire kit, untouched, onto another model, GLM-5. Its score on code tasks jumped from 58.5 to 75.8.

17.3 points. The gear wasn't tuned for it, and it took the gains for free.

This works because "gear"—the entire outer ring of prompts, tools, memory, skills, and orchestration logic—is a program, not muscle. Programs can be copied, given away, swapped around; muscle belongs to whoever grew it. The community calls this outer ring a harness; this article will just call it gear.

A World Without Answer Keys

How gear itself evolves is a harder problem. Plenty of prior work lets AI modify its own harness, but most of it lives in worlds with answer keys: math problems are either right or wrong, code either passes tests or doesn't. Real work tasks—writing a weekly report, booking a business trip—have no grader. You don't know whether a change actually improved things or whether you just got lucky this time.

Google Cloud AI Research's recent paper *Harness²* offers a method that is almost rustic in its simplicity: run controlled experiments. Three steps: propose, trial-run, combine. First, propose a few modifications to the current task's harness; then run the same task with both the modified and original harness under aligned conditions; finally, only accept changes supported by behavioral differences—the modified version must consistently perform better for the change to count.

With no grader, use "did behavior change after the edit" as evidence. A/B experiments replace homework grading; the paper calls this behavioral contrast.

Layer Two: The Gear That Builds Gear

Improving the current task's harness isn't enough—the next task starts from zero again. Hence the squared exponent in the title: a two-layer recursion, one wrapped around the task layer.

After a task finishes, the system writes successful changes, along with lessons about when and why they worked, into a library. Future tasks consult the library first and start with accumulated experience.

One discipline deserves its own note: experience only enters the library when supported by execution comparisons. A single good-looking output doesn't count—that could be luck. This threshold determines whether the library is an asset or a junk pile.

The results speak: a single contrast step gains up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench, and 11.8 percentage points on LAB; increasing the parallel-exploration budget from 1 candidate to 5 yields further gains. In ablation experiments where the experience library is removed—tasks still get done, but nobody reads anyone else's lessons—scores immediately fall, and the cost of reinventing the wheel on every task is priced in plain numbers.

The Gear-Gifting Experiment

Back to the opening numbers. The cross-model transfer experiments are the paper's hardest evidence: the harness tuned for Gemini 3.5 Flash, not changed by a single word, was fitted onto GLM-5, Gemini 3.7 Flash, and Claude Opus 4.8—and all three improved. GLM-5's 17.3 points on code tasks is larger than the gains many models get from tuning gear for themselves.

An even more striking finding: GLM-5—a weaker model—with upgraded gear outperformed Gemini with its stock gear. 75.3 vs 72.4. The gear gap is large enough to bridge a model-generation gap.

Together, these two results give the sentence worth stealing: gear is a program, muscle is weights. Programs can be gifted; muscle cannot.

A Product Line

This is the second paper from this research group in three weeks. Last month's RRSI (covered in our 09-26 anatomy: seven safeguards against harness evolution being skewed by data) handles settings with answer keys; this Harness² paper handles the open world without them. The same authors are relay-racing "harness evolution" from the lab to real tasks.

Costs must be stated plainly. Every propose-trial-combine step is real extra inference spend—each harness revision means re-running tasks, and the costs multiply. The paper itself admits: behavioral contrast doesn't guarantee changes transfer to other tasks; the experience library is currently just a simple index, and deduplication and staleness pruning at scale are left as future work.

The framework is open sourced under Apache 2.0, with 6,000+ stars on GitHub. One command runs it on your own tasks—harness2 --bench jb --k 1 --mode parallel. The ticket to harness evolution is, for now, this cheap.

---

*Sources: arXiv 2610.14224 (2026-10-09, Google Cloud AI Research, first author Ruiyao Xu, corresponding author Yanfei Chen); GitHub google-research/harness2 (6,107★, Apache 2.0); cross-model transfer and ablation figures from the paper body and Appendix E; RRSI = arXiv 2609.24972 (same group's prior work, 2026-09).*