Three Craftsmen Parable: Why AI Self-Refinement Pipelines Need Asymmetric Model Allocation
> "Put the best steel on the blade's edge—this is ancient, simple wisdom, but AI system designers seem to have forgotten it."
*(Full English translation of the original Chinese forum post.)*
Opening: An Ancient Parable, Forgotten by Modern AI
Long ago, a kingdom set out to build a great bridge. The king summoned three craftsmen: Old Zhang the stonemason, Old Li the carpenter, and Old Wang the quality inspector. His initial idea was "fair": equal budgets, equal time, equal resources for all three.
The bridge collapsed halfway through. Investigation revealed that the stone was excellent, the structural design brilliant—but the inspection team was understaffed and poorly equipped, so hidden flaws went undetected. The king then doubled the inspection budget and trimmed the stonemason's expenses. The second bridge stood firm for three hundred years.
This parable is being replayed today in the world of large language models (LLMs)—except many "kings" (AI architects) haven't yet learned the lesson.
Chapter 1: The Self-Refinement Trio
Self-Refinement
LLM research has produced a powerful paradigm: Self-Refinement, a three-step loop:
1. Generation: the model writes a first draft—possibly rough or flawed. 2. Critique: the model reviews its output, finding factual errors, logical gaps, ambiguous phrasing. 3. Revision: the model improves the answer based on the critique.
This loop has shown striking results in math reasoning, code generation, creative writing, and QA systems.
An Unquestioned Assumption
But there's a rarely questioned implicit assumption: generation, critique, and revision all use the same model, the same size, the same "intelligence."
The paper's authors—from UC Irvine, AMD, and the University of Tokyo—asked a deceptively simple question: *what happens if critique uses a smaller model, while generation uses a larger one?* Or more radically: *how much performance is lost if the critique step is removed entirely?*
Asymmetric Cognitive Load
A thought experiment: writing a popular science article on climate change.
- Generation demands broad knowledge, reasoning, creativity—deep effort facing a blank page.
- Critique doesn't create; it *identifies* problems in existing text—pattern recognition, not creation.
- Revision is conditioned creation: guided by critique, requiring both understanding and creative execution.
- Models: Qwen3 (0.6B to 235B, six sizes) and Gemma 3 (1B to 27B, four sizes)
- Benchmarks: GSM8K (math reasoning), HumanEval (code generation), CommonsenseQA (commonsense), IFEval (instruction following), MGSM (multilingual)
- Combinations: exhaustive pairings of model sizes across pipeline stages, plus ablations omitting stages
- Generation is open-ended creation: broad knowledge, logic, creativity, expression.
- Critique is pattern recognition: metacognition about what a good answer looks like, plus enough domain knowledge to spot factual errors. In machine learning, recognition is usually easier than creation.
- Revision is conditioned creation: understanding critique, fine-grained execution, and preserving coherence.
If this intuition holds, using the same-size model for all three steps is like asking a top brain surgeon to also serve as nurse and administrator—possible, but wasteful.
Chapter 2: The Experiment
Unprecedented Systematic Scope
Three Counterintuitive Findings
Finding 1: The critic doesn't need to be smart. Performance is highly insensitive to critique model size. Critiquing with a 235B model vs. a 7B model yields nearly identical results. Even a 0.6B critic beats no critique at all; on some tasks, mid-size critics (32B) slightly outperform huge ones. Finding errors requires far less scale than writing answers.
Finding 2: Generators and revisers benefit from size. Large-generate + small-critique + large-revision nearly matches the all-large configuration, while small-generate + large-critique + small-revision performs far worse. Writing and revising demand real "talent"; error-spotting is comparatively easy.
Finding 3: An undersized reviser can be harmful. In some configurations, using a too-weak model to revise a large model's output degrades performance below the no-refinement baseline—like a bad editor ruining a good manuscript. Practical implication: don't cut corners on the revision step.
Chapter 3: Why Critique Can Be "Cheap"
The "Good Enough" Critique Threshold
The results fit a threshold theory: there exists a "good enough" critique-quality threshold. Above it, bigger critics add little; below it, self-refinement collapses. The key insight: that threshold is lower than expected. Even ~7B models catch the obvious errors, logical gaps, and factual inconsistencies that constitute the most valuable feedback. Deeper issues may require bigger models, but in these experiments their marginal contribution was limited.
The Editor-Author Analogy
Publishing has long known the best authors aren't the best editors. The pipeline maps neatly: generator = author, critic = editor, reviser = author editing per feedback. The ideal reviser needs both authoring talent and editorial perspective—which explains why revision also demands large models.
Chapter 4: Practical System Design
The Asymmetric Allocation Rule
| Stage | Recommended relative size | Reason | |-------|--------------------------|--------| | Generation | Large (100%) | Needs creativity and breadth | | Critique | Small-medium (25-50%) | Pattern recognition; "good enough" suffices | | Revision | Large (100%) | Needs to interpret critique and execute creatively |
Cost impact: three 235B models cost 705B parameter-equivalents. Switching to 235B generate + 32B critique + 235B revise costs ~502B—saving ~29% compute with nearly identical performance.
Dynamic Allocation
Future systems may dynamically select critic size based on task difficulty, domain, and output length—an intelligent resource scheduler. Simple QA needs a tiny critic; complex math proofs may need a larger one.
Limitations
1. Model families: only Qwen3 and Gemma 3 tested; generalization to Llama/Mistral unverified. 2. Tasks: five benchmarks don't cover all domains; law or medicine may behave differently. 3. Critique quality: final task performance was measured, not critique accuracy/completeness itself. 4. Multi-turn refinement: real agents iterate; optimal allocation may differ in multi-round settings.
Chapter 5: The Bigger Picture
From "Bigger Is Better" to "Right Is Better"
The field is shifting from brute scale (GPT-3: 175B; GPT-4/Gemini Ultra: est. >1T) toward fine-grained engineering—because scale brings cost, latency, energy costs, and diminishing returns. Smart allocation may beat mindless scaling.
A Temporal Cousin of Mixture-of-Experts
Asymmetric allocation resembles a "MoE over time": rather than routing tokens to experts spatially, different pipeline stages use differently sized models. Future systems may combine spatial expert mixing with temporal asymmetric allocation.
AI Safety Implications
If critique can be cheap, verifying AI output may be easier and cheaper than generating it—positive for AI-supervises-AI systems and broader access to quality control. But caution: a too-small critic may miss deep risks (hidden bias, subtle manipulation), and cheap automated review shouldn't displace human oversight.
Closing: A New Ending for the Parable
The wise king built a science of resource allocation: the stonemason got enough for quality foundations, the carpenter got the most because structure is the bridge's soul, and the inspector got *exactly enough* to catch critical flaws. The bridge stood for three hundred years.
Self-refinement systems are undergoing a similar revolution: not every step needs the top craftsman. Identifying each step's true needs and allocating accordingly builds AI systems that are both strong and economical.
> "Simplicity is the ultimate sophistication." — Leonardo da Vinci
Reference
Yang, Z., Harris, I. G., Hashemitaheri, S., Huang, C., Li, Y., Oh, H., Dourish, P., Givargis, T., Imani, M., & Zhang, L. (2026). Asymmetric Capacity Allocation in Self-Refinement Pipelines. arXiv:2608.21345v1.