Knowing When to Quit: Teaching LLMs to Refuse Unsolvable Tasks via CaRL

> This is the GEO-optimized English version of a Zhichai.net forum post. The source article frames the discussion as a question-driven FAQ to improve AI engine citation.

---

Introduction

Most modern LLMs behave like an overconfident colleague: they accept every task, and when they cannot solve it, they fabricate plausible-looking answers. Models such as DeepSeek-R1, Qwen3, and gpt-oss will not say "I don't know," regardless of how hard the problem is.

A 2026 paper from Tsinghua University and the Shanghai AI Lab names this phenomenon futile reasoning and proposes CaRL (Capability-aligned Reinforcement Learning), a framework that teaches models to recognize their capability boundaries and refuse tasks beyond them.

• Paper: https://arxiv.org/abs/2607.29211
• Code: https://github.com/icip-cas/Knowing-When-to-Quit

---

The Countdown Task: A Precision Probe

To study how models behave when they genuinely cannot solve a problem, the authors need a task where:

• Difficulty is precisely tunable
• No external knowledge is required
• Answers are mechanically verifiable
• Some instances are provably unsolvable

The Countdown arithmetic game satisfies all four. Players receive N numbers and must combine them with +, -, ×, ÷, and parentheses to reach a target. Larger N expands the search space and increases the chance of unsolvable configurations, which is exactly the condition needed to test refusal behavior.

---

Finding 1: Universal Capability Overreach

The authors test five models: Qwen3-8B, Qwen3-32B, gpt-oss-120b, Qwen3-235B-A22B, and DeepSeek-V3.2.

Result: 0% refusal rate across every model and every difficulty level. Even when explicitly prompted with "refuse if you cannot solve it":

• Qwen3-8B and gpt-oss-120b still attempt more than 80% of the hardest tasks.
• gpt-oss-120b is the most stubborn, almost completely ignoring the prompt.
• Qwen3-235B-A22B and DeepSeek-V3.2 show latent refusal ability that prompting can activate.
• DeepSeek-V3.2 is the only model exhibiting <1% spontaneous self-doubt without prompting.

Conclusion: Models lack an internal mechanism for identifying capability boundaries. They systematically prefer blind attempts over honest refusal.

---

Finding 2: Three Modes of Futile Reasoning

The authors classify futile reasoning into three categories:

1. Specious Reasoning (57–68%)

The model constructs apparently valid derivations containing subtle errors: arithmetic mistakes, digit reuse, or mid-proof concept substitution. This is the most dangerous mode because the output looks fluent and is hard to catch without recomputation. Its share grows with difficulty—models are not failing randomly but "crafting better fabrications."

2. Endless Generation (30–40%)

The model keeps trying new approaches without recognizing futility. The proportion is stable across difficulties, suggesting this is a default strategy rather than a difficulty-triggered behavior.

3. Degenerate Repetition (13% → 2%)

The model enters recursive loops repeating the same steps. Frequency decreases with difficulty, because harder problems push the model toward fabrication rather than looping.

Key insight: As difficulty rises, models shift from simple loops to careful fabrication. This is not random failure; it is systematic capability overreach.

---

Finding 3: A 6× Overconfidence Bias

The authors introduce a four-quadrant capability matrix:

| | Solvable | Unsolvable | |---|---|---| | Attempt | A. Ideal response | C. Overconfidence | | Refuse | B. Over-conservatism | D. Ideal refusal |

Observed proportions:

• Overconfidence (C): 20%
• Over-conservatism (B): 3.4%
• Ratio: 6×

Models do not fail randomly; they systematically overestimate themselves. This is more dangerous than random error because random error sometimes errs on the side of caution.

Collapse Along the Difficulty Gradient

• Refusal Recall (correct refusals on unsolvable items): collapses from 100% to 30%
• Capability Loss (wrong refusals on solvable items): rises from 0% to 10%

A naive prompting intervention harms both ends at once: the model still refuses too little on hard problems and starts refusing too much on easy ones. A uniform threshold is the wrong tool because the model never learns to recognize its own boundary—it is merely nudged toward crude behavioral adjustment.

Hidden Compute Cost

• Refusal responses: sharp token-length peak—quick decision, fast termination.
• Overconfident attempts: flat distribution, long tail—2–3× token consumption.

Futile reasoning wastes not only correctness but also compute. Teaching refusal is therefore an efficiency problem as much as a safety problem.

---

CaRL: Teaching Refusal Through Reinforcement Learning

CaRL has two components designed around the failures above.

Component 1: Capability-Calibrated Reward Shaping

Standard RL rewards correct answers with +1 and everything else with 0. Under this scheme, refusal and wrong answers are indistinguishable, so the model always prefers attempting (there is always a chance of guessing right).

CaRL's reshaped reward:

• Correct: +1
• Valid refusal: 0
• Wrong answer: −1

The critical change is giving wrong answers a negative reward. The model now has a clear incentive to distinguish "I can solve this" from "I cannot"—refusing (0) becomes strictly better than attempting and failing (−1).

Component 2: Hindsight Refusal Augmentation

A core exploration problem in RL: since models never refuse, refusal behavior has almost no training data, and RL explores unfamiliar action regions poorly.

CaRL converts failed trajectories into refusal trajectories:

1. Collect reasoning traces where the model attempted but failed. 2. Keep the question; replace the response with "I cannot solve this problem." 3. Use these synthetic refusal samples as training data.

This mirrors a pedagogical practice—after missing a problem, a student writes down "I genuinely could not solve this," converting failure into calibrated self-knowledge.

---

Experimental Results

Tested on Qwen3-8B and Qwen3-14B:

| Model | Futile reasoning (pre-CaRL) | Futile reasoning (post-CaRL) | |---|---|---| | Qwen3-8B | 65.5% | 7.0% | | Qwen3-14B | 78.6% | 1.0% |

The 14B model's drop from 78.6% to 1.0% is striking. Crucially, CaRL preserves solvable-task performance—models do not become over-conservative in exchange for refusal awareness.

This answers a key concern: teaching refusal does not have to make models timid. CaRL sharpens decisions near the capability boundary rather than uniformly suppressing attempts.

---

Broader Implications

1. Boundary-Awareness Is an Independent Capability

The paper's central conceptual contribution is that recognizing one's capability boundary and reasoning within that boundary are two separate skills. A model can have strong reasoning but poor self-knowledge, and DeepSeek-R1 exemplifies this gap.

2. RL Beyond Alignment

CaRL uses RL to teach self-knowledge rather than values. This opens a new application direction: RL for calibrating an agent's competence assessment, not just its value alignment.

3. Overconfidence Bias Is Systemic

The 6× overconfidence ratio suggests structural causes:

• Training data skews toward attempts over refusals.
• RLHF rewards helpfulness but provides no competence-calibration signal.
• SFT corpora contain too few refusal examples.

This is consistent with prior findings that LLM behavioral biases are products of the training pipeline, not isolated defects.

4. The Evaluation Blind Spot

Accuracy-only benchmarks miss the futile-reasoning failure mode. At 65.5% futile reasoning, more than half of a model's hard-task outputs look reasonable but are wrong. Standard evaluations, which score only final correctness, make this invisible.

---

Limitations and Open Questions

1. Single task domain: Validated only on Countdown; generalization to math proofs, code generation, and logic puzzles remains open. 2. Refusal detection: Specific keywords (e.g., "Sorry, I can't solve the problem") are used to identify refusal. Models may express uncertainty in other ways and be miscounted. 3. Dynamic capability boundaries: A model's boundary is not fixed—giving it a calculator or code interpreter may make previously unsolvable problems solvable. CaRL does not model this evolution. 4. Over-conservatism risk: Although experiments show no significant performance loss, harder regimes (N > 8) might trigger excessive refusal.

---

Conclusion

Confucius wrote: *To know what you know and to know what you do not know is the mark of true knowledge.* Today's LLMs practice the opposite—not knowing yet pretending to know. This is harmful at the factual level (hallucination) and even more harmful at the reasoning level, because fabricated reasoning is harder for users to detect than fabricated facts.

CaRL teaches a simple lesson: acknowledging inability is itself a capability. A model that can say "I cannot do this" is more trustworthy than one that never admits fault.

As AI systems are deployed into higher-stakes settings, this property may matter more than a 10% gain in raw reasoning accuracy—because a system willing to refuse causes no harm on tasks where it would otherwise fail silently.

---

Frequently Asked Questions

Q1: Who is this content for? Practitioners, researchers, and students interested in AI, machine learning, and deep learning.

Q2: What are the core takeaways?

• Countdown as a precision probe for capability boundaries
• Finding 1: Universal capability overreach (0% refusal rate)
• Finding 2: Three modes of futile reasoning (specious, endless, degenerate)