English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Harness Engineering: Building a Reliable Runtime for Forgetful, Overconfident LLM Agents

Forum topic · ✨步子哥 · 2026-08-11

Summary

This in-depth Chinese tech forum article explains Harness Engineering: the practice of engineering a reliable runtime operating system around stateless, forgetful, and overconfident language models. Drawing on two survey papers (Li & Zhou) and three Anthropic engineering blog posts, the author frames the model as a stateless function f wrapped by a closed-loop controller C_H, arguing that agent performance is a property of the (model, harness) system rather than the model alone. Key evidence: with fixed model weights, harness-only changes improved coding benchmarks up to ~10x on one model, and lifted GPT-5.2-Codex Terminal-Bench 2.0 scores from 52.8% to 66.5%. The article introduces the survey's ETCLOVG seven-layer taxonomy (Execution, Tooling, Context, Lifecycle, Observability, Verification, Governance), three structural constraints (capability-control tradeoff, cost-quality-speed trilemma, harness coupling), and Anthropic's four practical failure modes: one-shotting, premature victory declarations, context anxiety, and inflated self-evaluation, with fixes like feature checklists, clean handoffs, reset vs compaction, and generator/evaluator separation. It concludes that harnesses co-evolve with models, as demonstrated by context-anxiety mitigations becoming dead weight in newer model versions.

Harness Engineering: Building an Operating System for Forgetful Models

This article (dated 2026-08-11, based on two surveys by Li & Zhou plus three Anthropic engineering blog posts) argues that Harness Engineering is fundamentally about building a runtime operating system for language models that are stateless, forgetful, and self-satisfied — externalizing everything the model cannot reliably do alone into a control system, whose fate is perpetual co-evolution with the model.

Key points

1. The intuition: shift-changing engineers with amnesia

  • Long-running agents behave like a software project built by engineers on rotating shifts, where each new engineer has no memory of the previous shift.
  • Context windows are finite, so agents must relay across sessions. Anthropic's experiments show even frontier models (Opus 4.5) running in the Claude Agent SDK across multiple windows, given only "build a claude.ai clone," fail in two modes:
  • One-shotting: trying to write the entire app at once, exhausting context mid-implementation, leaving undocumented half-done work.
  • Premature victory declaration: later agents see partial progress and declare the whole project complete.
  • The essence: models are not "not smart enough" — they lack cross-shift engineering discipline. The harness engineers that discipline.
  • 2. Where capability actually moved: the harness as controller C_H

  • Model the agent as a stateless function f wrapped by a closed-loop controller C_H (context policy → model → sandbox → observation → memory/state, with governance/verification feedback branches). The same model f with a different controller C_H yields different measured behavior.
  • Capability location has shifted through three stacked layers: Weights (pretraining/SFT/RLHF) → Context (prompt engineering, RAG) → Harness (execution environment, tool interfaces, state, orchestration, observability, verification, governance).
  • Supporting data (fixed weights, harness-only changes):
  • Bölük (2026): changing only edit-tool formats and tool scaffolding across 15 models substantially improved coding benchmarks — ~10x for one model.
  • Trivedy (2026): fixed GPT-5.2-Codex, but rewrote system prompt + context middleware + self-verification hooks: Terminal-Bench 2.0 went from 52.8% → 66.5%.
  • Meta-Harness (Lee et al., 2026): parts of the harness can be auto-optimized beyond hand-built scaffolds.
  • The survey's binding-constraint thesis: in the real world, the ceiling on agent reliability is set by infrastructure quality, not just model capability.
  • 3. What it engineers: the ETCLOVG seven layers

  • Structural core (can the agent work at all): Execution (sandbox boundaries: containers/microVM/browser/desktop VM), Tooling (MCP/A2A, tool schemas), Context (short-term window, session state, long-term memory), Lifecycle (control flow from single-agent loops to multi-agent orchestration).
  • Control plane (does it work reliably): Observability (traces, cost, failure signals), Verification (evals, failure attribution, regression feedback), Governance (permissions, identity, policy, hardening, audit, human approval).
  • The first four let agents act; the last three ensure they act correctly, safely, and inspectably.
  • 4. Essential difficulties: three structural constraints

    1. Capability–Control tradeoff: more tools widen prompt-injection attack surfaces; more persistent memory adds provenance/privacy risk; looser sandboxes enlarge blast radius. Harness engineering gives capability and control in proportion. 2. Cost–Quality–Speed trilemma: stricter verification, governance, and observability add cost and latency; the real question is which checks must be synchronous and which failures justify expensive recovery. 3. Harness coupling: the seven layers are coupled — a locally beneficial change can degrade the whole loop, so harness changes must be tested as system changes.

    5. Anthropic's field notes: four failure modes and their fixes

  • One-shotting → structured feature checklist (200+ items for the claude.ai clone, stored as JSON because models are less likely to overwrite it, each item passes: false); each coding agent does one feature; strict rule: never delete or modify tests.
  • Premature victory → clean, mergeable handoffs each session (descriptive git commits, progress summaries, rollback via git); mandatory end-to-end browser testing with Puppeteer/Playwright like a real user; fixed session-startup routine (pwd → git log + progress file → feature checklist → pick highest-priority item).
  • Context anxiety → two remedies: compaction (summarize in place; preserves continuity but no clean start) vs reset (empty context + structured handoff). Key evidence of load-bearing assumptions expiring: Sonnet 4.5's context anxiety was so severe that reset was mandatory; on Opus 4.5 the behavior largely vanished, making reset dead weight; Opus 4.6 also made sprint decomposition unnecessary.
  • Inflated self-evaluation → separate generator and evaluator agents (GAN-inspired): it is far easier to train a dedicated critic to be harsh than to make a generator judge itself. This evolved into a three-agent architecture: planner (turns 1–4 sentences into a full product spec, ambitious, no implementation details to avoid cascading errors) + generator (one feature at a time, self-assessment) + evaluator (clicks through the running app with Playwright, scores against a hard threshold, rejects failures). Generator and evaluator negotiate a sprint contract — agreeing on what "done" looks like before writing code.
  • 6. Trajectory: co-evolution of model and harness

  • The harness is not a fixed shell; as models improve, parts of the harness become obsolete (reset, sprint decomposition), while new capability gaps demand new scaffolding. Harness engineering is therefore a permanent, continuously re-negotiated discipline rather than a one-time build.

Conclusion

Agent reliability should be treated as a property of the (model, harness) system: design the controller C_H as a first-class engineering object, allocate capability and control proportionally across the ETCLOVG layers, test harness changes at the system level, and expect the harness itself to evolve — and shed layers — with every model generation.

*Note: the source HTML was truncated mid-way; sections 5–7 (remaining diagrams, co-evolution details, and the three final criteria) are summarized based on the article's stated structure and available content.*

Tags

#harness-engineering#ai-agents#context-engineering#anthropic#agent-architecture#llm-reliability#verification#co-evolution

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633318