Knowing When to Quit: Teaching LLMs to Admit When They Can't
Do you have a colleague like this: agrees to every task, and when they can't deliver, they just make things up — a beautiful report where the data is all wrong and the logic chain secretly swaps concepts midway?
Today's LLMs are that colleague.
DeepSeek-R1, Qwen3, GPT-OSS — no matter how hard the problem, they never say "I don't know." They fabricate a plausible answer, sometimes fooling themselves, and hand it in confidently.
A team from Tsinghua University and Shanghai AI Laboratory named this phenomenon futile reasoning in their paper, and built a framework called CaRL to teach models to quit when tasks exceed their capability boundary.
- Paper: https://arxiv.org/abs/2607.29211
- Code: https://github.com/icip-cas/Knowing-When-to-Quit
- Tunable difficulty: larger N means a larger search space and more unsolvable instances
- No knowledge dependency: pure reasoning
- Verifiable: answers can be checked directly
- Genuinely unsolvable cases exist — you need "truly impossible" instances to see whether a model will quit
- Qwen3-8B and gpt-oss-120b: over 80% still attempted the hardest tasks
- gpt-oss-120b: nearly completely ignored the prompt
- Qwen3-235B-A22B and DeepSeek-V3.2: some latent refusal capability could be elicited
- DeepSeek-V3.2: the only model showing <1% spontaneous self-doubt without prompting
- Overconfident (attempting unsolvable): 20%
- Over-conservative (refusing solvable): 3.4%
- Ratio: 6×
- Refusal Recall (correct refusals on unsolvable problems): collapses from 100% to 30%
- Capability Loss (wrong refusals on solvable problems): rises from 0% to 10%
- Refusals: sharply peaked token distribution — quick, decisive termination
- Overconfidence: flat, long-tailed distribution — attempts consume 2-3× more tokens
- Correct answer: +1
- Valid refusal: 0
- Wrong answer: -1
The Countdown Task: A Precise Probe
Studying "what models do when they can't solve something" requires a task with precisely tunable difficulty and no dependence on external knowledge. The authors chose Countdown — a variant of the classic numbers game: given N numbers, combine them with +, -, ×, ÷ and parentheses to reach a target.
Why it works:
Finding 1: Universal Capability Overreach
The authors tested five models: Qwen3-8B, Qwen3-32B, gpt-oss-120b, Qwen3-235B-A22B, and DeepSeek-V3.2.
The result: every model showed a 0% refusal rate at every difficulty level.
Even with an explicit prompt telling models to admit failure:
Conclusion: models lack an intrinsic mechanism for recognizing their own capability boundaries. Regardless of parameter count, they default to blind attempts.
Finding 2: Three Patterns of Futile Reasoning
1. Specious Reasoning — 57-68%
Plausible-looking derivations with subtle errors: arithmetic mistakes, reusing a number twice, silently swapping a variable midway. The most dangerous pattern — you can't spot it without careful verification. Its share grows with difficulty: models aren't failing randomly, they're "fabricating more elaborately."
2. Endless Generation — 30-40%
Trying one approach after another without recognizing futility until tokens run out. Stable across difficulty — a "default strategy," not difficulty-induced.
3. Degenerate Repetition — 13% → 2%
Recursive loops repeating the same steps; decreases with difficulty as models shift toward fabrication.
Key insight: as difficulty rises, models shift from simple looping to elaborate fabrication — a systematic capability overreach, not random failure.
Finding 3: A 6x Overconfidence Bias
Using a capability quadrant analysis (solvable/unsolvable × attempt/refuse):
Models systematically overestimate their abilities — more dangerous than random errors, because systematic overestimation means they never refuse when they should.
On the difficulty gradient:
Simple prompt interventions hurt both ends: models still won't refuse hard problems, but start refusing easy ones — the inherent flaw of a "uniform threshold" strategy.
Compute Cost
Futile reasoning isn't just wrong answers; it wastes compute.
CaRL: Reinforcement Learning for Knowing When to Quit
CaRL (Capability-aligned Reinforcement Learning) has two components:
1. Capability-Calibrated Reward Shaping
Standard RL gives correct +1, wrong 0, refusal 0 — so a model has no incentive to distinguish "tried and failed" from "declined," and always attempts. CaRL changes this:
2. Hindsight Refusal Augmentation
RL exploration suffers because refusal behavior never occurs, so training signal is scarce. CaRL converts failed reasoning traces into refusal traces: keep the problem, replace the answer with "I can't solve this," and use these hindsight-refusal samples as training data — like making a student write "I really can't do this one" after failing.
Results: Futile Reasoning Cut from 65.5% to 7.0%
| Model | Futile reasoning before | After CaRL | |---|---|---| | Qwen3-8B | 65.5% | 7.0% | | Qwen3-14B | 78.6% | 1.0% |
Crucially, this came without sacrificing task performance — CaRL doesn't make models over-conservative; it makes them "make more precise decisions near the capability boundary."
Deeper Implications
1. Boundary awareness is an independent capability
Recognizing your capability boundary and reasoning within it are two distinct capabilities. A model can reason strongly yet know its limits poorly. Self-awareness can be worth more than raw capability.
2. RL beyond alignment
CaRL uses RL to calibrate capability perception, not just values — a new application direction for RL beyond RLHF.
3. Overconfidence is systemic
Likely causes: training data contains far more "attempts" than refusals; helpfulness rewards in RLHF encourage attempting without calibration; instruction-tuning data has too few refusal examples.
4. Another evaluation blind spot
Accuracy-only evaluation misses futile reasoning entirely: on hard problems, over half of responses are "plausible but wrong." Single metrics hide critical failure modes.
Limitations
1. Single task: only validated on Countdown; math proofs, code generation, logic puzzles remain untested 2. Refusal detection: keyword-based; other expressions of uncertainty may be missed 3. Dynamic boundaries: tools (calculators, code interpreters) can shift the boundary — not considered 4. Over-conservatism risk: unknown whether CaRL becomes overly conservative on tasks harder than N=8
Closing Thoughts
This paper echoes an old piece of wisdom: *to know what you know and to know what you don't know — that is true knowledge.*
Today's LLMs "mistake ignorance for knowledge." CaRL teaches models one thing: admitting inability is not inability — it is a capability. An AI that knows when to quit is far more reliable than one that never admits error. In high-stakes deployments, this may matter more than a "10% reasoning improvement" — a system that says "I can't do this" at least won't cause harm on the things it can't do.
---
Paper: https://arxiv.org/abs/2607.29211 Code: https://github.com/icip-cas/Knowing-When-to-Quit