Scaling the Harness, Not Just the Model: Why AI Agent Systems Are the Next Bottleneck

This post is a detailed Chinese-language walkthrough of a recent UC Berkeley paper arguing that the next major bottleneck in agentic AI is not model scaling but system scaling — specifically, scaling the *harness*: the structured execution layer surrounding a foundation model.

> Paper: *From Model Scaling to System Scaling: Scaling the Harness in Agentic AI* > Author: Shangding Gu (UC Berkeley) > arXiv: 2605.26112 | GitHub: SafeRL-Lab/cheetahclaws > Categories: cs.AI, cs.LG | May 25, 2026

The Core Thesis

Over the past three years the AI industry has told a "model scaling" story: bigger models, more parameters, higher benchmark scores. The paper asks: once base models are sufficiently strong, why do agent systems still fail on long-horizon tasks? Its answer: we have been scaling the horse but forgetting the reins. The harness — tool interfaces, control loops, context construction, memory stores, skill routing, verification and governance — translates model "potential" into "long-horizon behavior." Once a model passes a capability threshold, the harness, not the model, becomes the primary determinant of real-world usefulness.

The Six-Component Framework

The paper formalizes harness quality as:

• R — Reasoning Substrate: the model itself (what model scaling upgrades)
• M — Memory Store: durable facts and past experience
• C — Context Constructor: decides what enters the model's context per turn
• S — Skill-Routing Layer: dispatches tasks to the right tools/sub-agents
• O — Orchestration Loop: the system's heartbeat, monitoring and adjusting execution
• G — Verification-and-Governance: the "immune system" checking actions and compliance

Model scaling only upgrades R; system scaling upgrades the other five.

Three levers operate at different time scales:

| Lever | Time scale | Role | Typical failure | |---|---|---|---| | Prompt | Local | Current goals/constraints | Fragile over long horizons; poor transfer | | Skill | Task-level | Reusable workflows | Mis-routing, bad composition | | Memory | Longitudinal | What persists over time | Drift, over-generalization, contamination |

Bottleneck 1: Context Governance — "Exposure Without Access"

Long context windows do not equal good context. Citing "Lost in the Middle" (Liu et al., 2024) and Gu (2026), the paper shows models preferentially attend to the beginning and end of long inputs, while relevant structure gets buried. Adding tokens indiscriminately often degrades performance. The proposed fix: treat context as the output of a selection policy — weight by semantic relevance, penalize redundancy, prioritize recently verified content, and record provenance for auditability. Claude Code's hybrid strategy (persistent CLAUDE.md guidance front-loaded, plus on-demand retrieval via glob/grep) is highlighted as good practice.

Bottleneck 2: Trustworthy Memory — "Stale-but-Confident"

Memory quality decomposes into four axes: precision, durability, retrievability, verifiability. The deadliest asymmetry: stale memories rarely block retrieval, but frequently drive confident actions on outdated assumptions (e.g., recalling a module path that no longer exists after refactoring). Al-Tawaha et al. (2026) show that memory-equipped agents carry *longitudinal* safety risks — more memory means more risk. The fix: make trust a runtime decision, not a stored property — apply staleness penalties, confidence thresholds, treat retrieved content as hypotheses requiring re-verification, and separate durable facts from environment-dependent ones.

Bottleneck 3: Dynamic Skill Routing — "Confident-but-Unchecked"

As specialized sub-agents proliferate, failures shift from "missing capability" to "capability without verification." Skill quality spans specificity, selectivity, composability, verifiability. Notably, Anthropic's internal research found that token usage alone explained 80% of performance variance in a multi-agent research system (95% with tool-call counts and model choice) — how a system allocates compute largely determines outcomes. Cemri et al. (2026) show multi-agent systems fail from design, misalignment, and insufficient verification — decomposition is easy; collaboration is hard. Routing should be a learned policy with confidence-aware escalation and postcondition checks as first-class citizens.

Comparing Three Real Systems

| Dimension | Claude Code | OpenClaw | CheetahClaws | |---|---|---|---| | Implementation | TypeScript | TypeScript | Python | | Scenario | Vendor-scale coding agent | Personal assistant | Research reference | | Context | User + project + session layers | User + channel-peer + session | User + project + session | | Memory | Persistent text, auto-extraction | Chat history + vector retrieval | Structured entries with confidence & recency | | Trust | Implicit (access patterns) | Implicit (usage frequency) | Explicit first-class fields |

Key finding: these systems solve the *same* core problems with different choices driven by deployment priorities, not model differences. Comparable base models project onto different harnesses yield qualitatively different agents.

Rethinking Evaluation

Current benchmarks (SWE-bench, AgentBench, WebArena, Terminal-Bench) conflate model and harness quality, report endpoint-only metrics, and reset state between runs. τ-bench (Yao et al., 2024) showed strong pass@1 agents can collapse under pass@k — a reliability gap. The paper calls for longitudinal evaluation of: memory retrieval precision, memory hygiene, minimal-context efficiency, communication fidelity, trajectory drift, verification-aware recovery, and safety under tool access. Agent quality should be measured as a longitudinal system property, not a one-shot completion score.

Answering Three Objections

1. *"Stronger models will internalize system management."* — Stale memories, over-broad tool permissions, and missing provenance are system failures; stronger models reduce their frequency but not the need for explicit mechanisms governing what is exposed, authorized, and audited. 2. *"End-to-end training will replace modular systems."* — Deployed agents touch private files, credentials, and external services; auditability, permissions, rollback, and provenance are security requirements, not engineering conveniences. 3. *"System-level evaluation is too expensive."* — Agents are deployed where cost, latency, tool risk, memory drift, and verification overhead determine usability; evaluation should expose these, not abstract them away.

Practical Takeaways for Individual Users

• Compare systems, not just models: examine context construction strategy, memory persistence, and tool design. Redesigning only the agent–computer interface (same model) substantially improved SWE-bench accuracy.
• Practice memory hygiene: regularly update or delete stale conventions; understand whether your tool derives trust implicitly or manages it explicitly.
• Multi-agent is not a silver bullet: focus on shared-state consistency, uncertainty-aware communication, contradiction detection, and task deduplication.
• Context management is your lever: write clear project guidance files, clean up resolved conversation branches, and specify task boundaries.

Conclusion

> "Future progress therefore requires system scaling: improving how agents construct context, maintain trustworthy memory, route skills, verify actions, govern tools, communicate across roles, and evolve over time."

The harness — not the model alone — is now the primary source of practical capability, and scaling the harness, in parallel with scaling the model, defines agentic AI's next major bottleneck.

References

1. Shangding Gu (2026). *From Model Scaling to System Scaling: Scaling the Harness in Agentic AI*. arXiv:2605.26112. GitHub: SafeRL-Lab/cheetahclaws 2. Liu et al. (2024a). *Lost in the middle: how language models use long contexts*. TACL 12, 157-173. 3. Yao et al. (2024). *τ-Bench*. arXiv:2406.12045. 4. Al-Tawaha et al. (2026). *Remembering more, risking more*. arXiv:2605.17830. 5. Cemri et al. (2026). *Why do multi-agent LLM systems fail?* NeurIPS 38. 6. Anthropic (2025d). *How we built our multi-agent research system*. https://www.anthropic.com/engineering/multi-agent-research-system 7. Kapoor et al. (2024). *AI agents that matter*. arXiv:2407.01502. 8. Jimenez et al. (2024). *SWE-bench*. ICLR 2024.