Knowing When to Quit: Teaching LLMs to Recognize Their Own Limits

LLMs today share a problematic trait: no matter how difficult the task, they rarely say "I can't." Instead, they generate plausible-sounding answers, subtly shifting concepts and quietly fabricating reasoning. A 2026 paper from Tsinghua University and Shanghai AI Lab calls this futile reasoning and proposes the CaRL framework to address it.

• Paper: https://arxiv.org/abs/2607.29211
• Code: https://github.com/icip-cas/Knowing-When-to-Quit

The Countdown Task as a Probe

The authors use the Countdown arithmetic puzzle, where the model must combine N numbers with addition, subtraction, multiplication, and division to reach a target. Key properties:

• Tunable difficulty: Larger N expands the search space and increases the chance of unsolvable instances.
• No knowledge dependency: Pure symbolic reasoning.
• Verifiable: Answers can be checked directly.
• Genuinely unsolvable cases: Some combinations truly cannot reach the target.

The last property is essential: without truly impossible tasks, you cannot test whether a model knows how to refuse.

Phenomenon 1: Universal Capability Overreach

Across five models (Qwen3-8B, Qwen3-32B, gpt-oss-120b, Qwen3-235B-A22B, DeepSeek-V3.2), the refusal rate was 0% at every difficulty level. Even when explicitly prompted with "if you cannot solve it, say so":

• Qwen3-8B and gpt-oss-120b still attempted over 80% of the hardest problems.
• gpt-oss-120b was the most resistant to prompting.
• Qwen3-235B-A22B and DeepSeek-V3.2 showed latent refusal ability activated by prompts.
• DeepSeek-V3.2 was the only model with any spontaneous self-doubt (<1%).

Conclusion: Models lack an intrinsic mechanism for identifying their capability boundaries.

Phenomenon 2: Three Modes of Futile Reasoning

1. Specious Reasoning (57–68%)

The most dangerous mode. The model constructs superficially coherent reasoning with subtle errors: arithmetic mistakes, number reuse, or covert concept substitution. As difficulty rises, this mode dominates — the model increasingly "crafts" rather than guesses.

2. Endless Generation (30–40%)

Stable across difficulty levels. The model keeps trying new approaches until tokens run out. This appears to be a default strategy.

3. Degenerate Repetition (13% → 2%)

The model falls into recursive loops. Frequency decreases with difficulty because hard problems push the model toward specious reasoning instead.

Key insight: As difficulty rises, models shift from simple loops to careful fabrication. This is systematic overreach, not random failure.

Phenomenon 3: A 6× Overconfidence Bias

Using a capability quadrant analysis:

| | Solvable | Unsolvable | |---|---|---| | Attempt | A. Ideal answer | C. Overconfidence | | Refuse | B. Overcaution | D. Ideal refusal |

Results:

• Overconfidence (C): 20%
• Overcaution (B): 3.4%
• Ratio: ~6×

Models systematically overestimate their abilities. Across the difficulty gradient, refusal recall on unsolvable problems collapses from 100% to 30%, while capability loss (rejecting solvable problems) rises from 0% to 10% — showing that naive prompting pushes the model in both directions simultaneously.

Overconfidence also inflates compute cost: overconfident responses consume 2–3× more tokens than decisive refusals.

CaRL: Capability-aligned Reinforcement Learning

The framework has two components.

Component 1: Capability-Calibrated Reward Shaping

Standard RL rewards: correct = +1, incorrect = 0, refuse = 0. Because trying-then-failing has the same reward as refusing, the model always prefers to attempt.

CaRL's reshaping:

• Correct: +1
• Effective refusal: 0
• Incorrect: −1

This gives the model an explicit incentive to distinguish "can solve" from "cannot solve," since refusal (0) beats a wrong attempt (−1).

Component 2: Hindsight Refusal Augmentation

Because models rarely refuse, refusal training data is scarce. CaRL converts failed reasoning trajectories into refusal samples by keeping the question and replacing the answer with "I cannot solve this problem," turning failures into refusal training data.

Results

| Model | Futile Reasoning (before CaRL) | Futile Reasoning (after CaRL) | |---|---|---| | Qwen3-8B | 65.5% | 7.0% | | Qwen3-14B | 78.6% | 1.0% |

Performance on solvable problems was preserved, confirming that the model is not simply becoming more cautious — it is making more precise decisions near the capability boundary.

Broader Implications

1. Capability-boundary awareness is a separate skill. A model can reason strongly while still lacking the metacognition to know when not to try. DeepSeek-R1 is a clear example. 2. RL extends beyond alignment. CaRL shows RL can calibrate a model's perception of its own abilities, not just enforce values. 3. Overconfidence is a systemic training artifact. Likely driven by training data skewed toward attempts, RLHF incentives, and SFT samples with few refusals. 4. Evaluation blind spots. Single accuracy metrics hide futile reasoning entirely. On hard problems, more than half of model outputs can appear reasonable while being wrong.

Limitations

• Validated only on Countdown; generalization to math proofs, code generation, and logic puzzles is open.
• Refusal detection relies on keyword matching, missing subtler forms of expressed uncertainty.
• Capability boundaries shift when tools (calculators, code interpreters) are introduced; CaRL does not model this.
• Whether CaRL becomes overcautious at very high difficulty (N>8) remains untested.

Conclusion

The paper's core lesson: acknowledging inability is itself an ability. As LLMs are deployed into higher-stakes settings, a system that says "I cannot do this" may matter more than one that is 10% better at reasoning — because it stops causing harm on tasks it cannot actually perform.

• Paper: https://arxiv.org/abs/2607.29211
• Code: https://github.com/icip-cas/Knowing-When-to-Quit