A Counterintuitive Observation
Within the Qwen2.5 family, which is more likely to make a mistake on a given math problem — the 1.5B small model or the 72B large model?
Intuitively, 72B is 48x larger and should be far less error-prone. But the CritICL paper uncovered something deeper: which problems the 1.5B model fails on, and what types of errors it makes, are highly consistent with the 72B model's.
Not the same error rate — the same error *shape*.
This sounds like bad news: large models inherit small models' flaws. But CritICL turns it into good news: since failure patterns are shared, small models' mistakes can serve as teaching material for large models.
Failure Modes: Not Just "Wrong," but "How It Went Wrong"
What is a "failure mode"?
A model answers a math problem incorrectly. The error itself carries no information — "it got it wrong" tells us nothing. But *the way* it errs carries a lot:
- Calculation errors: correct steps, broken arithmetic
- Logic jumps: skipping a key reasoning step
- Formula misuse: applying a similar but incorrect formula
- Condition omission: missing a constraint in the problem
- Overgeneralization: extending a special case to the general
- Termination failure: outputting an answer before reasoning completes
- CritICL-dynamic: up to 12.9% improvement
- CritICL-static: up to 13.4% improvement
- Paper: https://arxiv.org/abs/2608.27455
- HTML version: https://arxiv.org/html/2608.27455v1
- Code: https://github.com/umwyf/CRITICL
- License: CC BY 4.0
These are "failure modes" — type labels for errors. The CritICL team built a dataset called CritBank containing problems, small models' wrong answers, failure mode labels, and a critique for each failure mode.
A critique is not a generic "you're wrong," but "you misused a formula at this step; the correct formula is X." Like a teacher grading homework — not just a red cross, but margin notes.
Two CritICL Variants: Dynamic and Static
With CritBank in hand, how is it used? CritICL proposes two approaches.
CritICL-dynamic: Input-Aware Failure Mode Prediction
For each new question, first let the small model try it. Based on the small model's error pattern, predict what the large model might get wrong. Then retrieve critiques for that failure mode from CritBank and inject them into the large model's context.
The flow: new question → small model attempts → failure mode prediction → retrieve matching critique → large model answers with the critique in hand.
What does this resemble? An experienced test-taker reviewing past mistakes of the same type before an exam: "I stumbled here last time — watch out this time."
CritICL-static: Global Failure Modes
A simpler version: instead of per-question prediction, directly use critiques for the most common global failure modes as generic prompts.
Slightly less effective but cheaper — no need to run the small model online; just use a fixed critique set.
Why It Works: Cross-Scale Consistency of Failure Modes
CritICL's core assumption: within the same model family, small and large models share the same failure mode distribution.
The paper validates this in §4.1. Across the Qwen2.5 family (1.5B, 3B, 7B, 32B, 72B), the errors small models most frequently make are highly correlated with those of large models. If 1.5B tends to misuse formulas, so does 72B; where 1.5B makes logic jumps, 72B does too.
This consistency is the foundation of CritICL. If failure modes didn't transfer across scales, small models' errors would offer no guidance to large models.
Why does this consistency exist? The paper offers no theoretical explanation, but intuitively: models in the same family share architecture and training data recipes, so their learned blind spots are naturally similar. The large model just layers a correction on top of the small model's blind spots — but the blind spots themselves don't disappear.
Results: Up to 12.9% Improvement with Fewer Tokens
Tested on Qwen2.5-32B and 72B, on GSM8K, MATH, AMC, and AIME:
Compared to test-time scaling methods (repeated sampling, self-consistency, external verification), CritICL matches or exceeds accuracy while consuming significantly fewer tokens.
The reason is counterintuitive: CritICL increases input length (the injected critiques) but decreases output length. Because the model answers with an "avoidance guide," it doesn't need repeated trial and error — one pass yields a more accurate result. Total tokens are actually lower.
This aligns with the principle that "division of labor beats monolithic solutions" — the small model finds the pitfalls, the large model fills them, more efficiently than the large model groping on its own.
Cross-Family Transfer: Unexpected Good News
CritICL also ran a surprising experiment: can a CritBank built from Qwen-family small models help Llama-family large models?
Answer: yes, though less effectively than within the same family. Cross-family transfer partially works, meaning some failure modes are universal (common pitfalls of mathematical reasoning) and some are family-specific (training data biases).
This matters for real deployments: if you lack a same-family small model, another family's small model can still power a rough version of CritICL.
Limitations: Dependence on Family Structure
CritICL's hard constraint is the need for a same-family small model. If you're using a closed-source API (GPT-4, Claude), you can't access a same-family small model's internals, so CritICL doesn't apply.
Also, building CritBank requires an offline run of the small model: labeling failure modes and generating critiques. It's a one-time but nonzero cost. Appendix F.1 estimates this cost and concludes it's "acceptable" — CritBank is built once and reused.
Deeper Implications: Failure Is Structured
CritICL's real contribution isn't a specific technique but an insight: model failures are not random — they are structured.
This structure has two layers:
1. Type structure: failures can be categorized, each with a corresponding remedy 2. Scale structure: failure modes are consistent across scale within a family
Layer one makes "critique" possible — if failures were random, critiques would be meaningless. Layer two makes "small models teaching large models" possible — without cross-scale consistency, small models' errors would only help small models.
Together, these transform "failure" from "a bad thing to avoid" into "an exploitable information source." The small model is not a cheap substitute for the large one, but its diagnostician — telling you where the large model will stumble.
This complements the "evaluation blind spot law": that law says "what you don't test is where problems hide"; CritICL says "the failures you do measure are themselves teaching material." One is a warning; the other, an opportunity.
Paper and Code
CritICL's story recalls an old joke: novice programmers treat bugs as enemies; senior programmers treat bugs as lessons. CritICL applies this to models — a small model's bugs aren't shameful secrets to hide, but material for teaching the large model. Cross-scale failure pattern consistency means we no longer need to wait for the large model to err to discover problems; the small model has already flagged "this is where things will go wrong" in advance.
Division of labor beats unification: the small model finds the pitfalls, the large model fills them — cheaper and faster than the large model exploring alone.