Harness Scaling: Why the Agentic AI Bottleneck Has Moved from Models to Systems

This post discusses a UC Berkeley paper by Shangding Gu: "From Model Scaling to System Scaling: Scaling the Harness in Agentic AI" (arXiv:2605.26112, cs.AI/cs.LG; code: SafeRL-Lab/cheetahclaws).

Key points

The paper's central claim: the next major bottleneck in agentic AI is not model scaling but system scaling — improving the *harness*, the structured execution layer surrounding foundation models.

The harness as a six-component system

Agent performance is modeled as:

• R (Reasoning Substrate) — the base model itself; this is what model scaling upgrades.
• M (Memory Store) — persistent facts and prior experience.
• C (Context Constructor) — what gets selected into the model's context each turn.
• S (Skill-Routing Layer) — which tools/skills/sub-agents handle which subtasks.
• O (Orchestration Loop) — the control cycle monitoring and steering execution.
• G (Verification-and-Governance) — audits, permission checks, safety review.

Three levers operate at different time scales: Prompt (immediate goals; fragile over long horizons), Skill (reusable workflows; risks of mis-routing), and Memory (durable knowledge; risks of drift and pollution).

Bottleneck 1: Context governance

• "Lost in the Middle" (Liu et al., 2024) and related work show models attend unevenly to long inputs — long context ≠ good context.
• The paper calls this *exposure without access*: more tokens visible, but no proportional increase in attention to the right tokens.
• System-level fix: treat context construction as a selection policy — semantic relevance weighting, redundancy penalties, recency of verification, and source tracking. Claude Code's hybrid approach (persistent CLAUDE.md as priors plus on-demand glob/grep retrieval) is cited as a practical example.

Bottleneck 2: Trustworthy memory

Memory quality is decomposed into four axes:

\[\mathcal{M} = (\text{precision}, \text{durability}, \text{retrievability}, \text{verifiability})\]
• The key failure mode is stale-but-confident memory: outdated records are rarely blocked at retrieval, yet drive confident wrong actions (e.g., referencing a moved file path).
• Al-Tawaha et al. (2026) show memory-equipped agents carry longitudinal safety risks — more memory, more risk.
• Fix: make trust a runtime decision — staleness penalties at retrieval, confidence thresholds, treat retrieved content as hypotheses requiring re-verification, and separate durable facts from environment-dependent facts.
• CheetahClaws stores per-memory confidence and freshness as first-class fields; Claude Code and OpenClaw derive trust implicitly from access patterns.

Bottleneck 3: Dynamic skill routing

Skill quality spans specificity, selectivity, composability, verifiability. As specialization grows, failures shift from *missing capability* to *confident-but-unchecked* outputs. Anthropic's internal analysis found token usage alone explained ~80% of performance variance in a multi-agent research system (~95% with tool-call counts and model choice) — evidence that resource allocation, not raw model capability, dominates outcomes. Cemri et al. (2026) show multi-agent failures stem mostly from design, misalignment, and poor verification — decomposition is easy; coordination is hard.

System comparison: Claude Code vs OpenClaw vs CheetahClaws

| | Claude Code | OpenClaw | CheetahClaws | |---|---|---|---| | Scope | Vendor-scale coding agent | Multi-channel personal assistant (Discord/Slack/iMessage) | Research reference harness (Python) | | Memory | Persistent text memory, auto-extraction | Chat history + vector retrieval | Structured entries with confidence/freshness | | Openness | Closed | Open source | Open source |

Core finding: comparable models project onto different harnesses to produce qualitatively different agents — design choices are driven by deployment priorities, not model differences.

Rethinking evaluation

The paper argues endpoint benchmarks (SWE-bench, AgentBench, WebArena) conflate model vs harness gains and ignore cost, risk, and longitudinal behavior. τ-bench (Yao et al., 2024) shows agents strong at pass@1 can collapse at pass@k. Proposed additions: memory retrieval precision, memory hygiene, minimal-context efficiency, communication fidelity, trajectory drift, verification-aware recovery, and safety under tool access.

Rebuttals to three objections

1. *Stronger models will absorb the harness* — many deployment failures are system failures (stale memory, over-broad permissions) that better next-token prediction reduces but does not eliminate. 2. *End-to-end training replaces modularity* — deployed agents need auditable boundaries around private files, credentials, and tools; modularity is a governance requirement. 3. *System-level evaluation is too costly* — cost, latency, drift, and verification overhead are exactly what evaluation should expose.

Practical takeaways for users

• Compare harness design (context strategy, memory persistence, tool interfaces), not just model names — SWE-agent showed large gains from interface redesign alone with the same model.
• Practice memory hygiene: prune outdated conventions and verify recorded preferences.
• Multi-agent is not a silver bullet: focus on shared-state consistency, uncertainty communication, conflict detection, and task deduplication.
• Context management is the user-facing lever: write clear project guidance files, prune resolved conversation branches, specify task boundaries.

Conclusion

> Scaling the harness, in parallel with scaling the model, defines the next major bottleneck in agentic AI. For long-horizon agents, model capability alone is no longer a sufficient unit of analysis.

References: arXiv:2605.26112 (Gu, 2026); Liu et al. 2024 (Lost in the Middle, TACL); Yao et al. 2024 (τ-bench, arXiv:2406.12045); Cemri et al. 2026 (Why do multi-agent LLM systems fail?); Anthropic (multi-agent research system); Al-Tawaha et al. 2026 (arXiv:2605.17830); github.com/SafeRL-Lab/cheetahclaws.