Harness-1: Giving Search Agents an "External Brain" — How a 20B Model Beats Closed-Source Giants

TL;DR: Harness-1 frees search agents from juggling both "bookkeeping" and "reasoning." The environment side manages the candidate pool, evidence graph, and verification records, letting a 20B model focus purely on semantic decisions. Result: 73% average recall across 8 benchmarks, beating several closed-source frontier models, with transfer gains 2.2x larger than in-domain gains.

1. The Problem: Why Search Agents Run Out of "Brainpower"

Current search agents trained end-to-end force RL to optimize two fundamentally different things simultaneously:

• Semantic decisions: what to search, what to keep, what to verify, when to stop
• State management: maintaining candidate pools, importance ranking, cross-document link tracking, verification records

This leads to context bloat, sparse reward signals, and structural cross-document information buried in conversation flow. Harness-1's core insight:

> Any work that can be outsourced to the environment shouldn't burden the policy model.

2. Core Innovation: State-Externalizing Harnesses

Rather than making the model smarter, Harness-1 makes it more focused — replacing the chat stream with a structured state machine:

• Policy model (20B): decides what to search, keep/discard, verify, and when to stop
• Harness state machine (environment side) maintains:
• Candidate Pool: all search results, compressed via Sentence-BM25 (top-4 sentences) and deduplicated via MinHash
• Curated Set + importance labels: max 30 documents with four levels (very high / high / fair / low) — explicit curation instead of implicit top-k truncation
• Evidence Graph: a lightweight entity-document bidirectional map revealing bridge documents and singleton entities
• Verification Records: the model proposes claims, the harness records supported / unsupported / uncertain verdicts
• Budget Marker: injects [Token usage: 14,203/32,768] each turn, creating budget-aware decisions

Ablations confirm each mechanism matters: removing importance labels drops recall 4.1%, removing the evidence graph -2.6%, removing the verify tool -3.1%, and disabling all of them -12.2% — showing synergistic effects rather than independent features.

3. Training: SFT + RL on the Harness

Stage 1 — SFT: 899 high-quality trajectories generated by strong models (e.g., Kimi K2.5), filtered for trajectory recall > 50% and output recall > 40%. The goal is behavioral priors (parallel tool calls, query decomposition, pruning decisions), not answers.

Stage 2 — RL (CISPO): on-policy RL from the SFT checkpoint, 128 queries × 8 rollouts per step, 80 steps (~82K rollouts). The reward combines curated-set quality, trajectory coverage, answer evidence, answer discovery, tool diversity (w_div = 0.15), an omission penalty, and a turn penalty.

Without the tool-diversity reward, the policy collapses into "search-only" behavior (diversity ~3.5, recall ~0.53); with it, diversity stabilizes at ~4.30 and recall reaches ~0.60.

4. Results: Dominating 8 Benchmarks

| Model | Size | Avg Recall | Type | |-------|------|------------|------| | Opus-4.6 | ? | 79.4% | Closed frontier | | Harness-1 | 20B | 73.0% | Open small model | | GPT-5.4 | ? | 70.9% | Closed frontier | | Sonnet-4.6 | ? | 68.8% | Closed frontier | | Kimi-K2.5 | ? | 64.7% | Closed frontier | | Context-1 | 20B | 61.6% | Open baseline | | Tongyi DR 30B | 30B | 60.3% | Open mid-size | | GPT-OSS-120B | 120B | 59.0% | Open large | | Search-R1 32B | 32B | 28.9% | Open mid-size | | Qwen3 32B | 32B | 21.6% | Open mid-size |

The Striking Finding: Transfer

RL training used only one benchmark (SEC financial reports), yet on 4 unseen zero-shot benchmarks (LongSealQA, Seal0QA, FRAMES, HotpotQA) gains were +17.0 pts vs. the Context-1 baseline — versus +7.9 pts in-domain, a 2.2x transfer ratio. The reason: the policy learns operations over a domain-general search state. "Add this document to the curated set as high-importance" is the same action in patent search and web multi-hop QA.

5. Data Efficiency: 4K Queries vs. 221K for Competitors

• SFT: 899 filtered trajectories
• RL: 4,352 unique training queries

Compare: Search-R1 used 8K; Qwen3 used 221K. The efficiency comes from moving behavioral priors into the interface itself — RL optimizes semantic decisions on top of pre-installed behavior patterns.

6. Honest Limitations

• Narrow task scope: built for evidence retrieval (needle-in-haystack / multi-hop), not open-ended report generation
• Simplified evidence graph: regex-based entity extraction; no true entity disambiguation (e.g., "马云" vs "Jack Ma" won't be linked)
• LLM verifiers err: a wrong verification verdict can cause confident convergence on bad evidence
• Compression loss: Sentence-BM25 top-4 sentences may drop discourse-critical context, though read_document exists
• Ethics: no built-in access controls for sensitive, copyrighted, or biased content

7. Takeaway: The Interface Is Part of the Method

As the paper puts it: *"The interface itself is part of the method."* Instead of throwing massive data at a general-purpose model, design a structured interface that offloads the easy parts to the environment and lets the model learn only the hard parts. The same philosophy extends to code agents (environment maintains AST/symbol tables), data analysis agents (schema and statistical summaries), and diagnostic agents (medical timelines).

Harness-1 proves that retrieval capability is a function of interface design, not model size: a 20B model with a well-designed harness beats a 120B open model and multiple closed-source frontier models.

8. Conclusion

Harness-1 is one of the most solid pieces of engineering research in AI search: 73% average recall over 8 benchmarks, stronger transfer than in-domain performance, all with a 20B model and ~4K training queries. For multi-hop document retrieval in legal, financial, research, or compliance settings, the architecture is worth studying; for privacy-conscious deployments, the 20B model can run locally.

Links:

• Paper: https://arxiv.org/abs/2606.02373
• Code: https://github.com/pat-jj/harness-1
• Chroma Context-1 report: https://www.trychroma.com/research/context-1
• Related paper "Storage Is Not Memory": https://arxiv.org/html/2605.04897v1