Deep Dive: Comparing Open-Source Python Agent Harnesses
What is an Agent Harness?
Following Addy Osmani's widely-cited definition — Agent = Model + Harness — the harness is everything outside the model: the agent loop, tool definitions and execution, context management (trimming, compaction, subagent isolation), permissions and sandboxing, extension mechanisms (MCP, hooks, skills), and resumability.
The layered consensus (Winder.AI / LangChain): Model (reasoning) → Harness (runs one agent) → Framework (orchestrates multiple agents) → Platform (long-running, cross-team). In short: *frameworks compose agents; harnesses run them*.
Landscape
Two camps: the TS/Claude Code camp (Claude Code, OpenCode, Pi, Codex, Goose) ships polished terminal products; the Python camp (OpenHands, mini-swe-agent, smolagents, Pydantic AI, AgentScope) ships programmable execution layers with reproducible evaluation, closely tied to academia (Princeton, Berkeley).
The fall of first-generation Python coding agents (2023 class)
| Project | Outcome | |---|---| | gpt-engineer (55k★) | Archived; founder moved to Lovable | | GPT Pilot (33.7k★) | Abandoned for the commercial Pythagora.ai; hit by a supply-chain worm in Aug 2025 (credential-stealing commit, undetected until June 2026) | | Open Interpreter (68k★) | Python version dead; repo became a Rust fork of OpenAI Codex | | aider (48.7k★) | Effectively dormant; kept alive by the community fork aider-ce |
Lesson: the "code generator" form factor was displaced by the "agent harness" form factor. Post-2025, missing any of tool-loop, permissions, context engineering, or subagents means being out.
Active Python players (as of 2026-09)
| Project | Positioning | Stars | License | Status | |---|---|---|---|---| | OpenHands (software-agent-sdk) | Full-featured coding harness SDK | 86k | MIT | Active | | mini-swe-agent | Minimalist coding harness | 6.9k | MIT | Active | | smolagents | Minimal CodeAct agent library | 29.1k | Apache-2.0 | Slow maintenance | | AgentScope 2.0 | Claude Code-style harness framework | 30.6k | Apache-2.0 | Most active (daily commits) | | Pydantic AI 2 + pydantic-ai-harness | Typed framework + harness layer | 19.7k / 0.9k | MIT | Frequent releases | | OpenAI Agents SDK (Python) | Multi-agent SDK | 29.2k | MIT | 0.x iteration | | Qwen-Agent | Qwen companion framework | 17.1k | Apache-2.0 | Slowing since mid-2026 | | SWE-agent | Research harness (ACI pioneer) | 20.2k | MIT | Maintenance mode | | Agent-S | GUI automation agent | 12.2k | Apache-2.0 | Slow maintenance | | Letta | Memory agent platform | 24.6k | — | Moved to TypeScript |
Notably excluded: claude-agent-sdk-python is an MIT-licensed SDK shell bundling the proprietary Claude Code CLI — a closed harness with an open SDK skin.
Project Profiles
OpenHands V1 (software-agent-sdk) — the most engineered full harness
Core innovation is event sourcing: agents are stateless, immutable, serializable config objects emitting structured events via on_event(); all interactions are immutable events appended to a log, enabling deterministic replay and checkpoint resume — the only production-grade recoverability design in this comparison. The stateless Condenser swaps overflow events for summaries (default LLMSummingCondenser cuts API costs ~2x). Tools are Pydantic-defined Action–Execution–Observation; MCP is first-class (with OAuth). Sandboxing spans Local/Docker/K8s; a SecurityAnalyzer grades actions with human confirmation for high-risk ones. SWE-bench Verified: 72.8% (Sonnet 4.5). Weaknesses: repo fragmentation and identity drift toward "Agent Canvas," plus legacy configuration complexity.
mini-swe-agent — the minimalist standard-bearer
Princeton's rebellion against SWE-agent's complexity: the agent core is ~100 lines. Three radical decisions:
1. No tools except bash — it doesn't even use the model's tool-calling API (any model works)
2. No stateful shell sessions — every step runs via independent subprocess.run; sandboxing is just docker exec
3. No condenser — linear context growth, betting on long context windows
The message history exactly matches what is sent to the LM, making trajectories ideal for fine-tuning/RL baselines. Scores >74% on SWE-bench Verified; 76.8% with Claude Opus 4.5 — comparable to or better than far more complex harnesses. Users include Meta, NVIDIA, IBM, Anyscale.
SWE-agent — the ACI pioneer (maintenance mode)
Its NeurIPS 2024 Agent-Computer Interface paper (~2500 citations) established that interface design is prompt design for LMs: 100-line file viewers, an edit command that refuses syntactically invalid writes, search results trimmed to filenames. Its own README declares it superseded by mini-swe-agent — Princeton experimentally proved its own complex route was over-engineered.
AgentScope 2.0 — closest to "open-source Claude Code"
Alibaba's framework, the most active in 2026. Its 2.0 toolkit bundles Bash/Grep/Glob/Read/Write/Edit plus task/plan tools — Claude Code's full toolset — with ReAct interruption/resume, automatic compaction, fine-grained permissions with HITL, six middleware hook layers, eight sandbox backends, MCP + Skill Hub, a FastAPI multi-tenant agent service, and a terminal console. launch_console(agent) yields a Claude Code-like terminal experience. Weaknesses: heavy abstraction layers; product polish lags Claude Code/OpenCode.
Pydantic AI 2 + pydantic-ai-harness — the capability paradigm
v2.0 stable landed 2026-06-23. Key innovation: capabilities — instructions, tools, hooks, and settings bundled as a unit, loadable on demand (defer_loading=True), isomorphic to Claude Code Skills. The official pydantic-ai-harness provides a Coder() with filesystem path-traversal protection, shell command whitelisting with LLM API key env-stripping, AGENTS.md loading, planning, and context controls. Code Mode batches N tool calls in one Python script inside the Monty sandbox. Durability is a unique selling point: official Temporal/DBOS/Prefect integrations for cross-process crash recovery, plus Logfire (OTel) observability.
smolagents — CodeAct minimalism
HuggingFace's ~1000-line library where LLMs write Python as each action (CodeAct, from arXiv:2402.01030 — up to ~20% accuracy gains). Composable, token-efficient; sandboxes via E2B/Docker/Pyodide. Lacks persistence, memory, and compaction; releases slowing.
OpenAI Agents SDK — platform SDK turning harness-shaped
Flat Runner loop with handoffs, built-in tracing, Sessions persistence, Temporal durability. A key April 2026 pivot added Sandbox agents (SandboxAgent, Manifest, GitRepo) for long-horizon coding — absorbing Codex harness technology. Weaknesses: unstable 0.x API, best experience tied to OpenAI.
Others
- Letta (ex-MemGPT): three-tier self-editing memory + sleep-time compute (arXiv:2504.13171) are its legacy, but Python development is terminated in favor of TypeScript letta-code.
- Qwen-Agent: backs Qwen Chat in production; 1M-token RAG and Docker code interpreter; a model companion, not a harness.
- Agent-S: GUI automation; 72.6% on OSWorld with bBoN — claimed first superhuman computer-use result (Dec 2025).
- nanobot (HKUDS): ~4000-line Python minimalist harness.
- "The Scaffold Effect in Coding Agents" (arXiv): OpenHands-SDK 69% vs Goose/OpenCode at 38% with the same model
- Terminal-Bench 2.0: the same Opus model ranks up to 25 places apart depending on harness
- 99/100 of SWE-bench Verified leaderboard entries are self-reported
- Counter-example: Letta Code's persistent memory scored 59.1% on HAL vs Claude Code's 41.6% on the same Anthropic models
- OpenHands / software-agent-sdk · V1 path blog · SDK paper arXiv:2511.03690
- mini-swe-agent · SWE-agent · ACI paper arXiv:2405.15793 · HAL
- AgentScope · Pydantic AI v2 · pydantic-ai-harness
- smolagents · CodeAct paper arXiv:2402.01030 · OpenAI Agents SDK
- Addy Osmani: Agent Harness Engineering · Mario Zechner: Pi · Anthropic: Claude Agent SDK · Anthropic: Effective harnesses for long-running agents · Simon Willison: Designing agentic loops
- Winder.AI harness comparison · Scaffold Effect paper · Agent-S · Letta sleep-time compute
Cross-Cutting Architecture Comparison
Tool systems — three routes converging: JSON tool calling (mainstream), CodeAct / code-as-action (smolagents, OpenHands, Pydantic AI Code Mode — a view Anthropic's engineering blog also endorses: "code is precise, composable, infinitely reusable"), and bash-only minimalism (mini-swe-agent).
Context management — five schools: stateless compressors (OpenHands Condenser), automatic compaction (AgentScope / Claude Code), explicit history processors (Pydantic AI), no compression (mini-swe-agent), and structured persistence (aider's repo-map; planning-with-files' task_plan/findings/progress files — 26.5k★, evidence that context engineering is decoupling into portable plugins).
Sandboxing: most complete — AgentScope (8 backends) and OpenHands (SecurityAnalyzer + confirmation policies); minimal viable — mini-swe-agent's subprocess isolation. Winder.AI's lesson: sandboxing the filesystem doesn't cap billing or network blast radius.
The Harness Is the Biggest Performance Variable
The study's most important horizontal finding: the same model can score 5-30 percentage points differently across harnesses:
This motivated Princeton's HAL (Holistic Agent Leaderboard) for controlled harness comparison. Implication: any "model X achieves Y%" claim must first ask which harness; when building agents, engineering the harness may yield more than swapping models.
Minimalist vs full-featured verdict (unresolved): minimalism matches on benchmarks; full-featured harnesses retain a moat in event recovery, approvals, and memory for long-horizon production work. The stronger the model, the thinner the harness should be.
Selection Guidance
Six dimensions: form factor, model coupling, context-engineering autonomy, durability, security model, and evaluability (mini-swe-agent's clean trajectories are best for RL/fine-tuning loops).
| Scenario | Pick | Why | |---|---|---| | Learning harness internals / hacking | mini-swe-agent | ~100-line core; every line is a design decision | | Production agent product | OpenHands SDK or AgentScope 2.0 | Deepest engineering (event sourcing) vs most Claude Code-like and most active | | Enterprise typed + durable backends | Pydantic AI 2 + harness | Official Temporal/DBOS; newest capability abstraction | | Rapid prototyping / research | smolagents | CodeAct token efficiency, HF ecosystem | | RL/fine-tuning data loops | mini-swe-agent | Trajectories identical to LM input | | GUI/computer automation | Agent-S | Superhuman OSWorld result, no peer | | Qwen ecosystem | Qwen-Agent | Template-level fit + 1M RAG, accept slow updates | | Closed core for out-of-box use | claude-agent-sdk-python | Beware: closed harness + commercial terms |
Three heuristics: compare presuppositions, not feature lists (features have converged); plan on an 18-month horizon given the field's churn; own the durable parts (sandboxing, permissions, audit) since compaction and retry logic get obsoleted by model upgrades.
Closing: Three Trend Predictions
1. Convergence on code-as-action: smolagents, OpenHands, Pydantic AI Code Mode — even Anthropic concurs; JSON tool calling's dominance is eroding. 2. Thinner cores + on-demand capabilities: stronger models mean thinner loops (mini/pi), while skills/deferred capabilities turn heavyweight features into plugins. 3. Python's niche is evaluation, training, and SDK infrastructure: the TS camp won the terminal product; the Python camp is turning harnesses into reproducible, trainable, embeddable infrastructure. Letta's exit shows this niche isn't safe, but OpenHands SDK, AgentScope 2.0, and Pydantic AI advancing in parallel suggest the Python harness war is just entering its main event.