Harness Engineering: Building a Runtime Operating System for Forgetful, Overconfient Language Models
> TL;DR. Harness Engineering engineers a runtime operating system that lets stateless, forgetful, overconfident language models work reliably. Its core action is not "adding capability to the model," but externalizing—into a surrounding control system—everything the model cannot reliably do on its own. Its destiny is an unending co-evolution with the model.
---
01 · An intuition: the shift engineer with amnesia
The fastest way to understand *why* a harness is needed comes from an Anthropic metaphor: imagine a software project staffed by shift engineers, and each new engineer on shift has no memory of the previous shift.
This is the real situation of a long-running agent. The context window is finite, complex tasks cannot be completed in one window, so the agent must hand off across multiple *sessions* / context windows. Each new context window is a new engineer arriving on shift with amnesia.
A counter-intuitive fact: compaction alone is not enough. In Anthropic's experiments, even a frontier coding model (Opus 4.5), running in a loop across multiple context windows inside the Claude Agent SDK with the single instruction "build a clone of claude.ai," could not produce a production-grade app. It collapsed in two modes:
1. One-shotting. It tried to write the whole application in one go, ran out of context halfway, and left a pile of half-finished pieces with no documentation. The next shift could only guess—and burned most of its time getting basics working again. 2. Premature victory. After some functionality was in place, the next agent looked around, saw "there's already progress," and declared the whole project done.
The underlying truth: the model isn't "not smart enough"—it lacks a cross-shift engineering discipline. Harness Engineering is the engineering of that discipline.
---
02 · Where the capability really moved: the harness as controller
Harness as closed-loop controller $C_H$
Treat the model as a stateless function $f$, and the harness as the controller $C_H$ wrapped around it. The harness orchestrates: *task → context strategy → model f → action (sandbox) → observation/feedback → memory/state*, with verification and governance cross-cutting the loop.
Key point. With the *same* model $f$, change the controller $C_H$, and the measured behavior changes. Edit a context strategy, a tool schema, a verifier, or a recovery loop—you are editing $C_H$. Agent performance is a property of the (model, harness) system, not of the model. Harness Engineering treats the controller $C_H$ as a first-class engineering object.
weights, context, harness
The survey *Agent Harness Engineering* (Li & Zhou) traces where the community has been placing "intelligence":
| Layer | Capability is viewed as a property of | Engineering effort goes into | |-------|---------------------------------------|------------------------------| | Weights | model parameters | pretraining / SFT / RLHF | | Context | what the model sees at each step | prompt engineering / RAG / context engineering | | Harness | the infrastructure the model runs on | execution, tool interfaces, state, orchestration, observability, verification, governance |
The three layers stack, they don't replace each other—but each shifts where developers place "variable intelligence." Harness Engineering is the outermost layer's natural consequence.
Hard data, with fixed model weights and only the harness varied:
- Bölük (2026). Across 15 models, only changing edit-tool format + tool scaffolding produced large coding benchmark gains; one model improved by roughly 10×.
- Trivedy (2026). Fixed GPT-5.2-Codex; only rewrote the system prompt, injected a context middleware, and added self-verification hooks. Terminal-Bench 2.0 went 52.8% → 66.5%.
- Meta-Harness (Lee et al., 2026). A portion of the harness can be automatically optimized and even surpass handcrafted scaffolding.
- E · Execution. Where it runs; sandbox boundaries (container / microVM / browser / desktop VM).
- T · Tooling. How external capabilities are described, discovered, invoked (MCP / A2A / tool schemas).
- C · Context. Short-term window / session state / long-term memory—what the model sees per step.
- L · Lifecycle. Control flow: single-agent inner loop → multi-agent orchestration → full "issue → PR" pipeline.
- O · Observability. Traces, cost, failure, reliability signals.
- V · Verification. Turning tasks and traces into evaluations, failure attribution, regression feedback.
- G · Governance. Permissions, identity, policy, hardening, audit, human approval.
- Compaction. Summarize the old conversation in place; same agent continues. Preserves continuity but no clean starting point, anxiety persists.
- Reset. Clear context + structured handoff to the next agent. Clean starting point, at the cost of handoff overhead.
- Planner. 1–4 sentences → a complete product spec. Encourages ambition; owns product and high-level technical direction; does *not* lock down implementation details (errors otherwise cascade).
- Generator. One feature at a time, self-evaluates.
- Evaluator. Drives the app with Playwright like a user, scores each item, rejects below a hard threshold.
- A model with weaker long-context reasoning needs more aggressive reset, richer progress files, more verifiers.
- A model with stronger instruction following makes some guards redundant.
- A model with new tool-use conventions forces tool-schema and lifecycle rewrites.
- A model with new self-critique tendencies changes the generator/evaluator split's economics.
This is the binding-constraint thesis: in the real world, the reliability ceiling of an agent is set by infrastructure quality, not by model capability alone.
What it engineers: the ETCLOVG seven layers
The survey proposes a seven-layer taxonomy. The first four are *structural pillars*; the last three are a *control plane*.
Structural core (4 layers) — whether the agent can run
Control plane (3 layers) — whether the agent runs reliably
> The first four layers let the agent "do things"; the last three make sure it does them correctly, safely, and inspectably.
---
03 · The hard part: three structural constraints
Knowing *what* a harness is does not yet capture the essence. The essence hides in three structural constraints (the survey calls them *cross-layer problems*).
1. Capability–Control Tradeoff
The more capability you give an agent, the bigger the control problem. A wider tool menu → larger error-selection and prompt-injection surface. Stronger persistent memory → provenance, staleness, privacy risk. A looser sandbox → automation is more useful, but the blast radius of a mistake or compromise is larger. Harness Engineering isn't "give capability"—it's "with each increment of capability, give a proportional increment of control."2. Cost–Quality–Speed Trilemma
Stronger verification, stricter governance, richer observability, more faithful execution environments all add cost and latency. The real engineering question is not "the higher the quality the better," but deciding which checks must be synchronous, which can be asynchronous, and which failures are worth an expensive recovery.3. Harness Coupling — local optima are fragile
The seven layers are coupled: change the execution environment, evaluation results shift with it; tool descriptions consume context budget and shape model behavior; traces only become governance evidence when recorded at the same granularity as identity and permissions. A change that looks beneficial in an isolated test can make the whole loop worse when assembled. So harness changes must be tested as system changes.---
04 · Anthropic's field notes: four failure modes the harness fights
Theory is done. The most valuable part of Anthropic's three engineering blogs is *what* the harness concretely fights against and *how*. The harness engineers the daily discipline of a mature engineering team that the shift-engineer-with-amnesia lacks.
Failure mode ①: one-shotting → feature checklist + increments
Have an initialization agent expand the user's high-level prompt into a structured feature checklist (the claude.ai-clone example yielded 200+ items), each marked passes: false. Each subsequent coding agent does one feature at a time. Store the checklist as JSON rather than Markdown—models are less likely to tamper with or overwrite JSON. Add a hard rule: "modifying or deleting test cases is forbidden."
Failure mode ②: premature victory → clean handoffs + self-verification + human-like tests
Every session must end in a clean, mergeable state on main (no major bugs, ordered, documented): git commits with messages plus a progress summary, so the model can roll back bad changes with git. Never allow "mark complete without testing"—use browser automation (Puppeteer / Playwright MCP) to end-to-end test like a real user. Each new session begins with a fixed boot: pwd → read git log + progress file → read feature checklist → pick the highest-priority unfinished item.
Failure mode ③: context anxiety → compaction vs. reset
Models wrap up early as they approach the context limit (*context anxiety*). Two responses:
> Crucial evidence (and the cleanest example of "the load-bearing assumption goes stale"): Claude Sonnet 4.5's context anxiety is so strong that compaction is insufficient and reset is mandatory. Run the *same* harness on Opus 4.5 and that behavior mostly disappears—reset becomes dead weight you can delete. Then Opus 4.6 lets you delete the sprint decomposition that had previously been necessary.
Failure mode ④: inflated self-evaluation → split the generator from the evaluator
An agent scoring its own work will confidently praise it—even when a human sees mediocrity at a glance. Borrowing the GAN intuition, split the generator and the evaluator into two agents: train an independent evaluator to "love picking holes"—that is far more tractable than making the generator harsh on its own work. Once external feedback exists, the generator has a concrete iteration target.
This evolved into a three-agent architecture:
Before each sprint the two negotiate a sprint contract: first agree on "what 'done' looks like for this slice," then write code.
---
05 · The qualitative leap: the harness grows into an operating system
Once the seven layers are taken together and the three structural constraints are respected, the harness stops being "a wrapper around a prompt" and becomes something else: a runtime operating system for a non-deterministic, stateless compute unit.
The analogy is precise. An OS does not "give the CPU new instructions"; it gives the CPU deterministic execution, protected memory, device access, scheduling, and observability, so the CPU can do useful work. A harness does the same for the model: deterministic-enough execution (sandbox + tool interface), structured memory (C/L layers), device access (T), scheduling (L), observability (O), and verification + governance (V/G).
Three properties follow.
1. The harness is more durable than any single model. When Sonnet 4.5 was retired for Opus 4.5 on the same harness, *the harness barely changed*; the reset logic went from mandatory to dead weight, then was deleted. The infrastructure outlives the worker it manages. 2. It concentrates complexity outside the model. A model that cannot remember across shifts still becomes reliable—if everything it would have forgotten is encoded into state, context, verifiers, and recovery loops. 3. It shifts the unit of optimization. The interesting object is no longer "a smarter model" but "a better-tuned (model, harness) system." Meta-Harness shows this object can even be optimized automatically.
---
06 · Destiny: co-evolution between model and harness
Harness Engineering is not a one-time scaffolding that "completes" once the model is smart enough. It is in continuous co-evolution with the model—because each model generation changes which parts of the harness are load-bearing and which are dead weight.
---
07 · Coda: three takeaways and three judgment criteria
Three layers of closure.
1. Definition. Harness Engineering is engineering a runtime operating system around stateless, forgetful, overconfident models. 2. Structure. ETCLOVG — four structural layers plus three control-plane layers, governed by three structural constraints. 3. Destiny. Co-evolution with the model — perpetual re-tuning, never "done."
Three judgment criteria for any "agent" claim:
1. Layer test. Does the system explicitly handle E, T, C, L, O, V, G? 2. Constraint test. For each capability added, is a proportional control added? Is the cost–quality–speed trade-off explicit? Are changes tested as system changes, not local ones? 3. Co-evolution test. Is there a mechanism to retire harness pieces that the new model no longer needs—and to discover new failure modes the new model introduces?
If the answer to all three is yes, the agent is being engineered rather than merely prompted.
---
*Sources: Li & Zhou, "Agent Harness Engineering: A Survey"; Anthropic engineering blogs on Claude Agent SDK and long-running coding agents; Bölük 2026; Trivedy 2026; Lee et al., Meta-Harness 2026.*