Harness Engineering's Academic Formalization: From Industry Practice to Research Paradigm

LLMs can write poetry, program, and pass the bar exam—yet they fail at everyday tasks that take ten steps to complete. This is not because models are not smart enough, but because they lack something: a system that lets them keep working continuously. The industry calls this the Harness (scaffold/harness).

Over the past two years, companies like Anthropic and OpenAI have developed methodologies through engineering practice. In May 2026, researchers from Renmin University of China, Beijing University of Posts and Telecommunications, and other institutions published a survey paper that, for the first time, formally establishes Harness Engineering as an academic research paradigm. The paper's significance lies not in proposing a new algorithm, but in doing one thing: organizing scattered engineering experience into a system that can be researched, evaluated, and iterated upon. This marks a quiet paradigm shift in agent research—from "building smarter models" to "building smarter systems."

1. The Essence of the Problem: LLMs' Structural Deficiency

The core interface of large language models is single-turn generation: you input text, it outputs text, and the conversation ends. Its internal state does not persist to the next round—so-called "memory" is merely reassembling conversation history into the input window.

This design works fine for chat. But real-world problems are iterative, stateful, and require correction. When writing code, you fix compilation errors; when doing research, you update hypotheses after reading a paper; when managing projects, last week's decisions affect today's priorities. LLMs systematically fail in these scenarios:

• Losing intermediate goals: at step five of a multi-step task, forgetting what step one aimed to achieve
• Mis-calling tools: wrong parameters, wrong timing, no adjustment after failures
• Ignoring environmental feedback: code doesn't run, yet it continues down the same path

The authors put it sharply: these failures are "not a gap in model training, but a structural mismatch between the LLM's single-turn generation interface and the stateful, iterative nature of the real world." No matter how strong a model is, if it can only reason in single turns, it can never handle long-horizon tasks.

2. Three Perspectives on Harness Engineering

The paper defines Harness Engineering at three levels, from narrow to broad:

Level 1: Structural decomposition. An agent system consists of two parts—the foundation model and the harness supporting it. The model handles understanding, reasoning, and generation; the harness handles tool invocation, context maintenance, state persistence, and execution control. A model without a harness is a passive text generator; embedded in a harness, it becomes an agent capable of acting in real environments.

Level 2: Framework construction. The harness is not just a thin wrapper around the model, but complete, reusable software infrastructure supporting multi-step interaction, tool integration, and complex workflows. This level corresponds to agent frameworks (e.g., LangChain, AutoGPT).

Level 3: Co-augmentation (the paper's core perspective). Harness Engineering is not one-directional "building scaffolds for models" but bidirectional joint optimization:

• Scaffold side: designing more effective workflows, memory architectures, skill libraries, multi-agent orchestration
• Model side: through context engineering and agentic training, enabling models to better exploit scaffold capabilities

The paper emphasizes: "Both are indispensable. The most refined scaffold cannot save a model lacking basic reasoning ability; the strongest model cannot reach its potential in a poorly structured system."

3. Three Generations of Harnesses

Generation 1: Action Interface. The harness translates model output into executable actions and feeds environmental observations back. ReAct is representative—alternating "thinking" and "acting." This solved a basic problem: turning models from "speakers" into "doers."

Generation 2: Workflow Infrastructure. The harness manages persistent workspaces and orchestrates multi-step development processes. Models can view code, modify files, run commands, read test results, and iterate. SWE-agent, OpenHands, and Claude Code belong here. The key insight from SWE-agent: task performance depends not only on code generation quality but on how the harness organizes navigation, feedback, and iterative correction.

Generation 3: User-Centric Persistence. The harness transcends single tasks, managing user history, preferences, routines, and ongoing tasks for cross-session, cross-channel continuity. OpenClaw is representative—treating the assistant as a persistent, user-centric agent with memory, profile management, and delegation.

The pattern: harness boundaries keep expanding—from "connecting model to environment" to "orchestrating workspaces" to "maintaining user relationships." An agent's capability ceiling depends on how much the harness covers.

4. The Four Core Components

4.1 Agent Workflow

The workflow is the harness's operational core, turning a stateless LLM into a goal-directed system. Since the LLM terminates after one forward pass, "persistent behavior" is constructed entirely by the harness—it maintains message history and replays it each call, forming a loop: call model → parse response → execute tools → feed results into the next round.

This loop produces three tightly coupled submodules:

• Environmental perception: how the model "sees" current state
• Task planning: how to break big goals into executable steps
• Action execution: how to convert plans into concrete tool calls

4.2 Memory Systems

• Short-term memory: context maintenance within the current task—conversation history, intermediate results, execution traces
• Long-term memory: persistent cross-task knowledge—user profiles, skill accumulation, project history

Short-term memory solves "not losing focus"; long-term memory solves "remembering you."

4.3 Skill Libraries

Skills are reusable behavioral units, managed across three phases:

• Acquisition: extracting reusable operation sequences from task execution
• Management: indexing, retrieval, version control of skills
• Maintenance: retiring outdated skills, updating existing ones

Skill libraries let agents evolve from "relearning every time" to "what can be done can be remembered, and what is remembered can be used."

4.4 Multi-Agent Orchestration

When one agent is insufficient, the harness coordinates multiple agents via:

• Coordination architectures: hierarchical (supervisor-subordinate), peer-to-peer (equal negotiation), market-based (competitive bidding)
• Communication mechanisms: message passing, shared memory, blackboard systems

5. Model-Side Adaptation: Making Models Harness-Aware

Context engineering: designing better context structures and management strategies—including context design (organizing task descriptions, history, tool documentation) and context management (what to keep, discard, and compress in a limited window).

Agentic training: training models to acquire agentic capabilities, involving four elements:

• Environment construction: realistic training/evaluation environments (e.g., SWE-bench, WebArena)
• Training optimization algorithms: applying RL, imitation learning, DPO to agent scenarios
• Supervision signals: outcome feedback, process supervision, human annotation
• Infrastructure: compute frameworks and data pipelines for large-scale agent training

This chapter matters because it counters a common misconception: Harness Engineering is not just "systems engineering"—it also requires model-level innovation. The two co-evolve.

6. Five Evaluation Domains: A Capability Map

| Domain | Representative Benchmarks | Core Focus | |--------|--------------------------|------------| | Deep research | GAIA, WebArena, OpenResearcher | Long-horizon retrieval, multi-source integration, report generation | | Software engineering | SWE-bench, HumanEval, DS-1000 | Code understanding, editing, debugging, testing | | Tool use | ToolBench, APIBench, MINT | Tool selection, argument passing, error recovery | | Computer use | OSWorld, VisualWebArena | GUI understanding, web operation, visual grounding | | ML/scientific research | MLAgentBench, DiscoveryBench | Experiment design, data analysis, hypothesis testing |

The map reveals that different domains place vastly different demands on harnesses. Software engineering needs persistent workspaces and precise tool calls; deep research needs long-horizon memory and multi-source integration; computer use needs visual understanding and GUI operation. No "universal harness" fits all scenarios—one future direction is domain-specific scaffolds.

7. Six Future Directions

1. Efficiency: long-horizon tasks are computationally expensive; reducing reasoning rounds, compressing context, and optimizing tool calls are key to scale. 2. Safety: agents execute code, manipulate files, access networks—harnesses need built-in sandboxing, permission management, and audit logs. 3. Continual learning: most agents are "factory settings"; accumulating experience, updating skill libraries, and adapting to user habits remain open problems. 4. State and environment modeling: deeper harnesses need to understand causal structure, predict consequences, and build world models. 5. Embodied harnesses: physical-world agents (robots, autonomous driving) require handling sensor data, physical constraints, and real-time feedback—an order of magnitude harder. 6. Evaluation: existing benchmarks focus on final accuracy and lack systematic assessment of process reliability, robustness, and interpretability. The paper calls for treating "how it was done" as equally important as "what was done."

8. Conclusion: The Marker of a Paradigm Shift

The survey's value is not in any startling new technology, but in elevating Harness Engineering from industry experience to a researchable, teachable, evaluable academic paradigm. Its contributions can be summarized as three "ones":

• A unified vocabulary: Harness, Scaffold, Model-side Adaptation—terms that let researchers precisely discuss every layer of agent systems
• A systematic taxonomy: four components (workflow, memory, skills, orchestration) + two optimization directions (scaffold-side, model-side)
• A capability map: five evaluation domains + six future directions

When academia builds a complete taxonomy and evaluation framework for a concept, it ceases to be industry jargon and becomes a formal research field. The academic formalization of Harness Engineering marks a quiet but profound paradigm shift in agent research—from "model-centrism" to "system-centrism." The future agent race will belong not only to those with the largest models, but to those who best know how to scaffold them.

---

Reference paper: Xinyu Tang, Han Peng, Guoxin Chen, Yuze Shi, Zitao Su, Peiyu Liu, Wayne Xin Zhao, Yawen Li, Zhe Xue. "Agent Systems with Harness Engineering." OpenReview Archive, 2026-05-18. https://openreview.net/pdf?id=nM5tDHrQsx