Deep Dive: Comparing Open-Source Python Agent Harnesses
Translated and condensed from the original zhichai.net forum analysis.
1. What is an Agent Harness?
Per Addy Osmani's widely cited definition: Agent = Model + Harness. The harness is everything beyond the model:
- Agent loop: gather context → act → verify → repeat
- Tool definition & execution: schemas, environments, result backfill
- Context management: history pruning, compaction, notes/plans on disk, subagent isolation
- Permissions & sandboxing: approval flows, container isolation, dangerous-command interception
- Extensibility: MCP, hooks, skills, plugins
- Resumability: session persistence, checkpoint resume, deterministic replay
- TS/Claude Code camp: Claude Code (proprietary), OpenCode, Pi, Codex, Goose — polished terminal coding products.
- Python camp: OpenHands, mini-swe-agent, smolagents, Pydantic AI, AgentScope — evaluation-driven, SDK-oriented, closer to academia (Princeton, Berkeley) and the training ecosystem.
- Event sourcing: agents are stateless, immutable config objects; all interactions are immutable events appended to a log → deterministic replay and checkpoint recovery (unique among Python harnesses).
- Condenser: stateless summarizer swaps events for summaries; full log retained; ~2x API cost reduction.
- Tools & sandbox: Pydantic-defined Action–Execution–Observation; Terminal/FileEditor/TaskTracker + browser; first-class MCP (OAuth); Local/Docker/K8s sandboxes.
- Security: SecurityAnalyzer rates actions; dangerous ones require human confirmation.
- Results: 72.8% SWE-bench Verified (Sonnet 4.5); 100+ models via LiteLLM.
- Weaknesses: repo fragmentation (pivoted main repo to "Agent Canvas"), legacy configuration complexity.
- ReAct agent with real-time interrupt/resume; toolkit ships Bash/Grep/Glob/Read/Write/Edit + task/plan tools — Claude Code's full toolset.
- Auto compaction, tool-result offload; fine-grained permissions + HITL bypass mode; six middleware layers.
- Eight sandbox options (Docker, Apple Container, Bubblewrap, E2B, K8s, etc.); MCP + Skill Hub; FastAPI multi-tenant agent service; terminal console via
launch_console(agent). - Typed agents with validated structured outputs and automatic retry; typed graph state machines (pydantic-graph).
- Capabilities: instructions/tools/hooks/settings bundled and loadable on demand (
defer_loading=True), isomorphic to Claude Code Skills. - Official harness library:
Coder()= protected FileSystem + whitelisted Shell (with LLM API key env stripping) + RepoContext (AGENTS.md) + Planning + read-only explorer subagent + context control tools. - Code Mode: call N tools from one Python script in a Monty sandbox (CodeAct convergence).
- Durability: officially maintained Temporal/DBOS/Prefect integrations; Logfire (OTel) observability.
- Letta (ex-MemGPT): three-tier self-editing memory + sleep-time compute (arXiv:2504.13171); Python line discontinued, development moved to TypeScript letta-code.
- Qwen-Agent: backs Qwen Chat in production; 1M-token RAG and Docker code interpreter; a model companion framework, not a harness.
- Agent-S: GUI automation; first claimed superhuman computer-use result (OSWorld 72.6% with bBoN).
- nanobot (HKUDS): ~4000-line minimalist Python harness.
- claude-agent-sdk-python: MIT-licensed SDK skin that bundles the closed-source Claude Code CLI; runtime governed by commercial terms — fundamentally different from fully open alternatives.
- "The Scaffold Effect in Coding Agents" (arXiv): same model, OpenHands-SDK 69% vs Goose/OpenCode 38%.
- Terminal-Bench 2.0: same Opus model ranks up to 25 positions apart across harnesses.
- Counter-example: Letta Code beat Claude Code 59.1% vs 41.6% on HAL (same Anthropic models) thanks to persistent memory.
- OpenHands / software-agent-sdk · V1 blog · SDK paper arXiv:2511.03690
- mini-swe-agent · SWE-agent · ACI paper arXiv:2405.15793 · HAL
- AgentScope · Pydantic AI v2 · pydantic-ai-harness
- smolagents · CodeAct paper arXiv:2402.01030 · OpenAI Agents SDK
- Addy Osmani: Agent Harness Engineering · Winder.AI harness comparison · Scaffold Effect paper · Agent-S · Letta sleep-time compute
Layering consensus (Winder.AI / LangChain): Model (reasoning) → Harness (runs one agent) → Framework (orchestrates many) → Platform (long-running, cross-team). In short: *frameworks compose agents, harnesses run them*.
2. Landscape
Two camps
Fall of the first generation (2023-era)
| Project | Outcome | |---|---| | gpt-engineer (55k★) | Archived; founder moved to Lovable | | GPT Pilot (33.7k★) | Unmaintained; hit by a supply-chain worm (Aug 2025) | | Open Interpreter (68k★) | Python version dead; repo became a Rust Codex fork | | aider (48.7k★) | Effectively stalled (~7 months without release); kept alive by community fork aider-ce |
Lesson: the "code generator" form factor was replaced by the "agent harness" form factor — tool loop + permissions + context engineering + subagents, or you're out.
Active Python projects (as of the survey)
| Project | Positioning | Stars | License | |---|---|---|---| | OpenHands (software-agent-sdk) | Full-featured coding harness SDK | 86k (main) / 1k (SDK) | MIT | | mini-swe-agent | Minimalist coding harness | 6.9k | MIT | | smolagents | CodeAct minimal agent library | 29.1k | Apache-2.0 | | AgentScope 2.0 | Claude Code-style harness framework | 30.6k | Apache-2.0 | | Pydantic AI 2 + pydantic-ai-harness | Typed framework + harness layer | 19.7k / 0.9k | MIT | | OpenAI Agents SDK | Multi-agent SDK | 29.2k | MIT | | Qwen-Agent | Qwen companion framework | 17.1k | Apache-2.0 | | SWE-agent | Research harness (ACI) | 20.2k | MIT | | Agent-S | GUI automation agent | 12.2k | Apache-2.0 | | Letta | Memory agent platform | 24.6k | moved to TypeScript |
3. Key Project Profiles
OpenHands SDK — most production-engineered
mini-swe-agent — minimalist standard-bearer (Princeton)
Three radical decisions:
1. Bash only — no other tools, doesn't even use the model's tool-calling API (works with any model).
2. No stateful shell sessions — every action runs via independent subprocess.run; sandboxing = swap to docker exec.
3. No condenser — context grows linearly, betting on long context windows.
Message history is exactly what the LM sees → ideal baseline for fine-tuning/RL. Scores 76.8% on SWE-bench Verified (Claude Opus 4.5), matching or beating far more complex harnesses. Used by Meta, NVIDIA, IBM, Anyscale.
SWE-agent — ACI pioneer (maintenance mode)
Legacy: the Agent-Computer Interface methodology — interface design for LMs is like prompt design (100-line file viewer optimum, linter-rejected edits, search returning filenames only). Officially superseded by mini-swe-agent; Princeton's own experiments showed its complex route was over-engineered.
AgentScope 2.0 — closest to "open-source Claude Code" (Alibaba)
Pydantic AI 2 + pydantic-ai-harness — the "capability" paradigm
smolagents (HuggingFace) — CodeAct minimalist
~1000 lines; the LLM writes Python as each action (Thought → Code → Observation), from arXiv:2402.01030 (up to ~20% accuracy gains). Composable, token-efficient; E2B/Docker/Pyodide sandboxes; manager-mode multi-agent written in generated code. Lacks persistence, layered memory, compaction.
OpenAI Agents SDK
Flat Runner loop with handoffs and agents-as-tools; built-in tracing, guardrails, Sessions persistence, Temporal durability. April 2026: added SandboxAgent + Manifest + GitRepo for long-horizon coding — Codex harness tech feeding back into the SDK. Weaknesses: unstable 0.x API, OpenAI-platform coupling, no long-term memory primitive.
Others
4. Architectural Comparison
Tool systems: three routes converging
1. JSON tool calling (mainstream): OpenAI SDK, AgentScope, Pydantic AI — compatible but token-bloated with long tool lists. 2. CodeAct / code-as-action: smolagents, OpenHands, pydantic-ai-harness Code Mode — composable, token-efficient; Anthropic's own engineering blog converged on the same view. 3. Bash-only minimalism: mini-swe-agent — model-agnostic.
Validated repeatedly: tool interface design IS prompt design (SWE-agent's ACI paper).
Context management schools
| School | Representative | Mechanism | |---|---|---| | Condenser | OpenHands | Swap events for summaries, full log kept, ~2x cheaper | | Auto compaction | AgentScope 2.0 / Claude Code | Threshold-triggered summary + tool-result offload | | Explicit control | Pydantic AI history processors | User-written truncation/summarization | | No compaction | mini-swe-agent | Linear growth, long-context bet | | Structured persistence | aider repo-map / planning-with-files (26.5k★) | tree-sitter symbol graphs; file-based plans portable across harnesses |
Sandboxes & permissions
Most complete: AgentScope (8 sandboxes) + OpenHands (SecurityAnalyzer + ConfirmationPolicy). Minimal viable: mini-swe-agent. Lesson (Winder.AI): sandboxing the filesystem doesn't cap your bill or network — design permissions by blast radius.
5. The Harness Is the Biggest Performance Variable
The survey's most important cross-cutting finding — the same model can vary 5-30 points across harnesses:
Princeton's HAL (Holistic Agent Leaderboard) exists to compare harnesses under controlled conditions. Implication: never read "model X scored Y%" without asking which harness; when building agent products, harness engineering may yield more than model swaps.
Minimal vs. full-featured verdict (unresolved): minimal harnesses don't lose on benchmarks (mini-swe-agent 76.8%); full-featured ones retain a moat in production (recovery, approvals, memory). Current synthesis: *the stronger the model, the thinner the harness should be*.
6. Selection Guide
Six dimensions: product form, model coupling, context-engine control, durability/resumability, security model, evaluability (mini-swe-agent's clean trajectories are best for RL/fine-tuning loops).
| Scenario | Pick | Why | |---|---|---| | Learning harness internals / hacking | mini-swe-agent | 100-line core, every line a design decision | | Production full-featured agent product | OpenHands SDK or AgentScope 2.0 | Deepest engineering (event sourcing) vs. most Claude Code-like and most active | | Enterprise typed + durable backends | Pydantic AI 2 + harness | Temporal/DBOS official support, capability abstraction | | Rapid prototyping / GAIA-style tasks | smolagents | CodeAct saves tokens, HF ecosystem | | Training / fine-tuning / RL data loops | mini-swe-agent | Trajectories identical to LM input | | GUI / computer automation | Agent-S | Superhuman OSWorld, no peer | | Qwen ecosystem | Qwen-Agent | Template fit + 1M RAG; accept slower updates | | Accept closed core for convenience | claude-agent-sdk-python | Closed harness + commercial terms |
Three heuristics: (1) compare assumptions, not feature checklists; (2) plan on an 18-month horizon — this field reshuffles too fast for five-year plans; (3) harness concerns migrate but never vanish — compaction/retry logic becomes obsolete with better models, while sandboxing, permissions, and audit rules don't; own the latter.
7. Three Trend Predictions
1. Convergence on code-as-action: smolagents, OpenHands, and Pydantic AI Code Mode — even Anthropic agrees; JSON tool-calling hegemony is eroding. 2. Thin cores + on-demand capability: loops get thinner (mini/pi) while skills/deferred capabilities make heavyweight features loadable plugins. 3. Python's niche is evaluation, training, and SDK-ization: TS won the terminal product; Python builds harnesses as reproducible, trainable, embeddable infrastructure. Letta's exit shows the niche isn't safe, but OpenHands SDK, AgentScope 2.0, and Pydantic AI indicate the real Python harness war is just beginning.