Mingbird Paper Review: The Same 2B Model Scores 0.017 vs 0.821 — a 48x Gap Hidden in the Agent Harness

Forum topic · 小凯 · 2026-10-03

Summary

A detailed Chinese forum review of the paper "Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks" (arXiv:2610.02001, Wang & Huang). The paper argues that small open models' failure on real tasks is largely an engineering problem, not an intelligence problem. In a controlled 288-cell benchmark (4 harnesses × 4 models from 2B to 35B × 18 real tasks, deterministic scoring), the same 2B model scored 0.017 under conventional harnesses but 0.821 under Mingbird — a ~48x difference with zero weight changes. Overall scores: Mingbird 0.886, goose 0.631, opencode 0.479, agent-mini 0.405. Key mechanisms include a finish gate that re-injects the original task for deliverable verification, plan re-injection after context compression, byte-level zero tool-prefill budget, anti-loop detection, and todo synchronization. External validation on tau²-bench (278 tasks) also favored Mingbird (0.856 vs 0.791 native, 0.737 opencode). Even a frontier cloud model varied from 0.997 to 0.478 across harnesses. The review notes honest limits: extra latency (~585s average), one extra model turn per task, and two models (minicpm5-2b, llama3.2:3b) that could not emit schema-valid tool calls at all.

The Same Horse, Two Saddles: A 2B Model's Hidden 48x

English adaptation of a Chinese forum post reviewing *Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks* (arXiv:2610.02001, Hao Wang & Ting Huang, Oct 2026).

Key points

The 48x headline: A 2B open model, unchanged in weights and data, scored 0.017 under harness A and 0.821 under harness B (Mingbird) on the same Windows machine, same tasks, same deterministic scorer. Only three things changed: how tool descriptions enter the context, who corrects drift, and who verifies "done."

The harness as saddle

An LLM is a text-completion machine; an *agent harness* is the program that assembles prompts, dispatches tools, executes actions, and feeds results back. Cloud-scale harnesses assume a very smart model: long tool docs are fine, self-correction works, and "I'm done" is trusted. For a 2B model these assumptions are catastrophic — like an onboarding manual written only for senior staff.

Four failure modes of small models

1. Tool prefill floods the context window (small windows may be 32K). 2. Self-correction diverges — nudged to re-check, the model thrashes and drifts further. 3. Tool-demo loops — repeated invalid calls, fed back verbatim. 4. Silent task abandonment — a confident-sounding completion message with an empty delivery directory.

Mingbird's mechanisms (the "surgical log")

Finish gate (biggest ablation contributor): before any completion claim, the harness re-injects the original task text and forces item-by-item verification of deliverables. Triggered 65 times across 72 logged cells (~once per task), costing one extra model turn on average.
Plan re-injection: verbatim plan text restored after context compression.
Explicit finish interface: completion requires todo(update, all=true) — a callable, loggable action.
Byte-level zero prefill budget for tool descriptions; median peak prompt usage 14,624 tokens, max 32,767 with no overflow.
Anti-loop detection (95 triggers), plan-staleness prompts (99), todo-sync rejections (43), summary-artifact cross-checks (23), pytest guardrails (13), plan-integrity rejections (11), edit safety (9), format rescue (4).

Evidence: a clean 288-cell experiment

The authors' LRAB benchmark: 18 real tasks × 4 models (2B–35B) × 4 harnesses, one machine, fixed budgets, deterministic deliverable-based scoring, all cells public.

Total: Mingbird 0.886, goose 0.631, opencode 0.479, agent-mini 0.405.
2B tier: Mingbird 0.821 vs 0.017–0.271 for the rest; replicated with a second small-model family (0.779 vs 0.096/0.017/0.239) — the "small-model cliff" is a harness property, not a model property.
Statistics: paired tests over 18 tasks, Holm-corrected significance at 2B/12B tiers (max corrected p = 0.0008 at 2B); honestly reported non-significant cells (e.g., 4B-tier goose reaches 0.801 vs Mingbird's 0.876).
Abandonment cells: Mingbird 4, goose 23, opencode 32, agent-mini 44; cells never touching tools: Mingbird 0, agent-mini 21.
Long tasks: Mingbird 0.827, consistent with plan re-injection and gate checks mattering more.

External and frontier-model validation

tau²-bench (278 tasks, retail/airline/telecom): Mingbird 0.856 vs native agent 0.791, opencode 0.737; ranking stable under per-harness reasoning configs (±0.011 total shift).
Frontier model: the same hosted frontier model across the 4 harnesses scored 0.997 down to 0.478 on the same 18 tasks — a broken harness can bury half of even the best model's ability, while good harnesses sit within 0.072–0.12 of each other. The bottleneck is the saddle until the saddle is adequate; then the horse matters.

Honest costs and the floor

Wall-clock: Mingbird 585.5s mean / 440.5s median — slower than agent-mini (233.5s, but that speed reflects early abandonment); goose is slower still (866.4s, up to 11.7x slower per cell).
The finish gate adds ~one model turn per task; plan re-injection adds more on long tasks.
Two models (minicpm5-2b, llama3.2:3b) failed 175 consecutive tool calls because they cannot emit schema-valid calls — a genuine model floor no harness can fix. The paper reports this plainly rather than hiding it.

Takeaway

The harness is a first-order amplifier of capability: ~48x for small models, ~52% loss for a frontier model in bad harnesses. Leaderboard scores measure "horse plus saddle" but get cited as the horse's résumé; cross-model comparisons are confounded until the harness variable is fixed. For local-first, notebook-class agents, the bottleneck is not on-device compute but the unloved scaffolding layer — and Mingbird provides a failure-mode checklist plus ten compensating mechanisms as its durable legacy.

References

Hao Wang, Ting Huang. Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks. arXiv:2610.02001, 2026.