> Feynman once said: "A theory's elegance lies not in its complexity, but in how simply it explains complex phenomena." Today's finding flips this idea — sometimes, if you package a simple thing properly, it can unleash far more power than you'd expect.
---
Introduction: Small and Sharp, or Big and Clumsy?
Over the past two years, the AI narrative has been "scale is everything." GPT-4 reportedly has 1.7 trillion parameters; Claude, hundreds of billions. More parameters, more intelligence — so the story goes.
But a newly published experimental paper offers a counterintuitive conclusion: for operational tasks, a 2-3B parameter small model paired with a carefully designed "execution harness" can match or even beat a bare larger model. What matters isn't how big the model's "brain" is, but how good the "armor" you put on it is.
---
Chapter 1: Comparing Three Harnesses
The researchers designed a clever experiment. Same small model (2-3B parameters), three different harnesses:
Condition 1: Bare Model (Model-Only)
Directly give the model a prompt and expect correctly formatted output. No harness, no error correction — it's all on the model.Condition 2: Minimal Shell
Add a thin wrapper around the prompt — some XML tags marking input/output boundaries. One extra layer of format constraint beyond the bare setup.Condition 3: Four-Stage Pipeline
A carefully designed execution harness with four stages: 1. Plan: the model first "thinks through" what to do and outputs an execution plan 2. Execute: carry out the plan step by step 3. Verify: check whether results meet expectations and format requirements 4. Recover: if verification fails, automatically backtrack, correct, and re-execute---
Chapter 2: Unexpected Results
Test results across 24 operational tasks:
| Condition | Task Success Rate (TSR) | Valid Task Success Rate (VTSR) | |------|-------------------|------| | Bare model | 0.429 | — | | Minimal shell | lower than bare | — | | Four-stage pipeline | 0.952 | 1.000 |
Two key findings:
Finding 1: The Minimal Shell Made Things Worse
Most counterintuitive — adding the minimal shell (format tags) decreased the success rate. This is a "non-monotonic phenomenon." The implication: a half-done harness is worse than no harness. Either don't do it, or do it completely — the full four-stage pipeline.Finding 2: Small Model + Good Harness ≈ Bare Large Model
A Gemma4 2B with the four-stage pipeline hit a 95.2% success rate across 21 tasks, with a 100% rate of valid outputs (correct format + verifiable content). The bare model managed only 42.9% — and it wasn't just poor output quality; format itself collapsed — LLaMA 3.2 3B simply gave up on JSON under complex format requirements.---
Chapter 3: Why the Harness Matters More Than Size
"Scaffold Collapse"
The paper introduces the concept of Scaffold Collapse: when output format requirements are complex (e.g., returning structured JSON), a model without external harness support will abandon format constraints under high cognitive load — outputting free text and breaking the entire task pipeline.The four-stage pipeline solves this via its "verify + recover" stages: even if the model collapses at some step, the harness detects the format error, automatically backtracks, and lets the model regenerate.
Ablation: Each Stage's Contribution
The paper ran ablations to measure each stage's contribution:- Planning stage contributed ~24.7% of total gains
- Recovery stage also contributed ~24.7% of total gains
- Verification capture rate (VCR) = 0.625 — 62.5% of errors caught at verification
- Title: It's Not the Size: Harness Design Determines Operational Stability in Small Language Models
- Author: Yong-eun Cho
- arXiv: 2605.12129
- Core finding: A four-stage pipeline (Plan → Execute → Verify → Recover) brings small models to a 0.952 success rate
Planning + Recovery = half the gains. This shows the small model's weakness isn't "can't do it" — it's "can't plan the order of doing things" and "doesn't know how to fix mistakes."
---
Chapter 4: Engineering Philosophy — "The Model Is the Engine, the Harness Is the Chassis"
The paper's insight in one line:
> A large model running bare is like a car with a big-horsepower engine but no steering wheel or brakes — great on a straight line, but it crashes on every turn.
A small model with a good execution harness (planning, verification, recovery) is like a mid-sized engine with four-wheel drive, ABS, and lane-keeping — less horsepower, but far more stable, and much more likely to actually arrive.
This overturns a hidden assumption in AI: "for more reliability, use a bigger model." The paper's data says: not necessarily. For operational tasks (multi-step execution, format constraints, error recovery), harness design should take priority over scaling model size.
---
*Paper information*