A Counterintuitive Observation
In the Qwen2.5 family, who is more likely to make a mistake on a given math problem: the 1.5B small model or the 72B large model?
Intuition says the 72B — 48x larger — should be far less error-prone. But the CritICL paper uncovers something deeper: which problems the 1.5B model fails on, and what types of errors it makes, are highly consistent with the 72B model's failures.
Not the same amount of error — the same error "shape".
This sounds like bad news: even large models carry small models' flaws. But CritICL turns it into good news: since failure patterns are shared, small models' mistakes can serve as teaching material for large models.
Failure Modes: Not Just "Wrong", But "How Wrong"
A model answers a math question incorrectly. The error itself carries little information — "it computed wrong" is obvious. But the *way* it fails carries a lot:
- Calculation errors: correct steps, broken arithmetic
- Logic jumps: skipping a key reasoning step
- Formula misuse: applying a similar but incorrect formula
- Missing constraints: overlooking a condition in the problem
- Over-generalization: extending special-case conclusions to general cases
- Termination failure: outputting an answer before reasoning completes
- CritICL-dynamic: up to 12.9% improvement
- CritICL-static: up to 13.4% improvement
- Paper: https://arxiv.org/abs/2608.27455
- HTML version: https://arxiv.org/html/2608.27455v1
- Code: https://github.com/umwyf/CRITICL
- License: CC BY 4.0
These are failure modes — error type labels. The team built a dataset called CritBank, containing questions, small models' wrong answers, failure-mode labels, and critiques tailored to each failure mode.
A critique is not a vague "you're wrong" — it's "you misused a formula at this step; the correct one is X." Like a teacher marking homework not with an X, but with margin notes.
Two Variants: Dynamic and Static
CritICL-dynamic: Input-aware failure mode prediction
For each new question, first run the small model on it. Based on the small model's error pattern, predict what the large model might get wrong. Then retrieve critiques matching that failure mode from CritBank and inject them into the large model's context.
Pipeline: new question → small model trial → failure mode prediction → critique retrieval → large model answers with the critique in context.
It's like an experienced exam-taker reviewing past mistakes of the same type beforehand: "I stumbled here last time — be careful this time."
CritICL-static: Global failure modes
A simpler version: skip per-question prediction and use critiques for the globally most common failure modes as generic prompts.
Slightly weaker but cheaper — no need to run the small model online; just use a fixed critique set.
Why It Works: Cross-Scale Consistency of Failure Modes
The core hypothesis: within the same model family, small and large models share a failure mode distribution.
The paper validates this in §4.1. Across Qwen2.5 (1.5B, 3B, 7B, 32B, 72B), the small models' most common failures correlate strongly with the large models'. If the 1.5B tends toward formula misuse, so does the 72B; where the 1.5B makes logic jumps, the 72B does too.
This consistency is the foundation of CritICL. If failure modes didn't transfer across scales, small-model errors would hold no guidance for large models.
Why the consistency? The paper offers no theory, but intuitively: models in the same family share architecture and training data recipes, so their blind spots are naturally similar. The large model adds a corrective layer on top of the small model's blind spots — but the blind spots themselves don't disappear.
Results: Up to 12.9% Gains with Fewer Tokens
Tested on Qwen2.5-32B and 72B across GSM8K, MATH, AMC, and AIME:
Compared with test-time scaling methods (repeated sampling, self-consistency, external verification), CritICL matches or beats accuracy while consuming significantly fewer tokens.
The reason is counterintuitive: CritICL increases input length (the injected critiques) but decreases output length. With an "avoidance guide" in hand, the model doesn't need repeated trial and error — it gets closer to the right answer in one pass. Total tokens actually drop.
This aligns with the principle that division of labor beats unification: small models find the pits, large models fill them — more efficient than the large model exploring alone.
Cross-Family Transfer: An Unexpected Bonus
Another experiment: can a CritBank built from Qwen small models help Llama large models?
Answer: yes, though less effectively than same-family use. Cross-family transfer partially works, implying failure modes are partly universal (common math-reasoning traps) and partly family-specific (training data biases).
This matters for deployment: without a same-family small model, you can still build a rough version of CritICL using another family's small model.
Limitations: Dependence on Family Structure
CritICL's hard constraint is requiring a same-family small model. With closed-source APIs (GPT-4, Claude), you can't access a same-family small model's internals, so CritICL doesn't apply.
Also, building CritBank requires an offline pass: running the small model, labeling failure modes, generating critiques. It's a one-time cost, but not zero. Appendix F.1 estimates it as "acceptable" — build CritBank once, reuse repeatedly.
The Deeper Implication: Failure Is Structured
CritICL's real contribution isn't a specific technique but a realization: model failures are not random — they are structured.
Structured in two senses:
1. Typed structure: failures can be categorized, each with a corresponding correction strategy 2. Scaled structure: failure modes are consistent across scales within a family
The first makes "critique" possible — if failures were random, critique would be meaningless. The second makes "small models teaching large models" possible — if failure modes didn't transfer across scales, small-model errors would only help small models.
Together, these transform "failure" from "a bad thing to avoid" into "an exploitable information source." Small models aren't just cheap substitutes for large models — they are diagnostic instruments that reveal where the large model will stumble.
This complements the "evaluation blind spot" principle: blind-spot analysis says "untested areas hide problems"; CritICL says "the failures you find are themselves teaching material." The former is a warning; the latter an opportunity.
Paper & Code
CritICL recalls an old joke: novice programmers treat bugs as enemies; veteran programmers treat bugs as teaching material. CritICL applies this to models — a small model's bugs aren't shameful secrets to hide, but material for preparing the large model's lessons. Cross-scale consistency of failure modes means we don't need to wait for the large model to fail to find problems; the small model has already flagged "this is where things break" in advance.
Division of labor beats unification: small models find the pits, large models fill them — cheaper and faster than the large model groping on its own.