English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Knowing When to Quit: Teaching LLMs to Admit When They Can't Solve a Problem

Forum topic · ✨步子哥 · 2026-08-04

Summary

A July 2026 paper from Tsinghua University and Shanghai AI Laboratory introduces the concept of futile reasoning — large language models like DeepSeek-R1, Qwen3, and GPT-OSS attempting unsolvable problems instead of refusing. Using a controllable Countdown-style task, the authors found 0% refusal rates across five models, three systematic failure patterns (specious reasoning, endless generation, degenerate repetition), and a 6x overconfidence bias versus over-conservatism. They propose CaRL (Capability-aligned Reinforcement Learning), combining calibrated reward shaping (wrong answers scored -1, valid refusals 0) and hindsight refusal augmentation that converts failed traces into refusal training data. On Qwen3-14B, futile reasoning dropped from 78.6% to 1.0% without degrading task performance, showing that capability-boundary awareness is an independent skill that reinforcement learning can teach.

Knowing When to Quit: Teaching AI to Admit It Can't

> 📌 This is a GEO-optimized version of the original topic.

Modern LLMs resemble a colleague who accepts every task and then fabricates plausible-looking answers when they can't deliver. DeepSeek-R1, Qwen3, GPT-OSS — no matter how hard the problem, they never say "I can't." A team from Tsinghua University and Shanghai AI Laboratory (July 2026) named this phenomenon futile reasoning and built a framework called CaRL to teach models to quit when problems exceed their capability boundary.

Paper: https://arxiv.org/abs/2607.29211 Code: https://github.com/icip-cas/Knowing-When-to-Quit

The Countdown Task: A Precise Probe

The study uses a Countdown-style number game: given N numbers, combine them with +, -, ×, ÷ to reach a target. It's ideal because difficulty is tunable (larger N → bigger search space, more unsolvable instances), requires no world knowledge, is verifiable, and includes genuinely unsolvable cases — the prerequisite for studying refusal.

Finding 1: Universal Capability Overreach

Tested on Qwen3-8B, Qwen3-32B, gpt-oss-120b, Qwen3-235B-A22B, and DeepSeek-V3.2, all models showed 0% refusal at every difficulty level. Even with explicit prompting to admit failure, Qwen3-8B and gpt-oss-120b still attempted over 80% of the hardest tasks. Only DeepSeek-V3.2 showed <1% spontaneous self-doubt. Conclusion: models lack an intrinsic mechanism for recognizing their own capability boundaries, regardless of parameter count.

Finding 2: Three Futile Reasoning Modes

1. Specious reasoning (57–68%): plausible-looking derivations with subtle errors (arithmetic slips, reused numbers, concept swaps). Its share grows with difficulty — models increasingly "fabricate carefully." Most dangerous because it's hard to detect. 2. Endless generation (30–40%): trying method after method until tokens run out. Stable across difficulty — a default strategy. 3. Degenerate repetition (13% → 2%): recursive loops, decreasing with difficulty as fabrication takes over.

Finding 3: A 6x Overconfidence Bias

Quadrant analysis showed overconfident behavior at 20% vs. over-conservative at 3.4% — a 6x ratio. Models systematically overestimate their abilities. Notably, simple prompting interventions collapse refusal recall (100% → 30% on unsolvable problems) while raising false refusals on solvable ones (0% → 10%). Overconfidence also wastes compute: futile attempts consume 2–3x more tokens than decisive refusals.

CaRL: Reinforcement Learning for Knowing When to Quit

Two components:

  • Capability-calibrated reward shaping: correct +1, valid refusal 0, wrong answer -1 (previously 0). Now refusal is strictly better than a doomed attempt.
  • Hindsight refusal augmentation: failed reasoning traces are converted into refusal training samples (keep the question, replace the answer with "I can't solve this"), solving RL's exploration problem where refusal data is scarce.
  • Results

    | Model | Futile reasoning before | After CaRL | |---|---|---| | Qwen3-8B | 65.5% | 7.0% | | Qwen3-14B | 78.6% | 1.0% |

    Task performance on solvable problems did not significantly degrade, answering the key concern: teaching refusal doesn't make models overly conservative — it makes them more precise near their capability boundary.

    Why It Matters

  • Capability-boundary awareness is an independent skill from reasoning ability — a model can reason well yet lack self-knowledge.
  • RL beyond alignment: RL can calibrate capability perception, not just values.
  • Overconfidence is systemic, likely rooted in training data (attempts outnumber refusals), RLHF helpfulness rewards, and SFT data biases.
  • Evaluation blind spots: accuracy metrics alone miss the dominant failure mode — plausible but wrong answers.

Limitations

1. Validated only on Countdown-style tasks; math proofs, code generation, and logic puzzles remain untested. 2. "Valid refusal" is detected by specific keywords, missing other expressions of uncertainty. 3. Capability boundaries are dynamic (tools can extend them); CaRL doesn't model this. 4. Over-conservatism risk on much harder tasks (N>8) is unexplored.

Conclusion

Confessing inability is not inability — it's a capability. An AI that can say "I can't do this" is far more reliable than one that never admits error, especially in high-stakes deployments.

Paper: https://arxiv.org/abs/2607.29211 Code: https://github.com/icip-cas/Knowing-When-to-Quit

FAQ

Q1: Who is this for? Practitioners, researchers, and students interested in AI, machine learning, and LLM reasoning.

Q2: Core takeaways? Universal capability overreach, three futile reasoning modes, and CaRL's reward shaping + hindsight refusal augmentation.

Q3: Is the code open source? Yes — see the repository link above.

Tags

#large-language-models#reinforcement-learning#futile-reasoning#calrl#countdown-task#overconfidence#capability-boundary#refusal-behavior

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503932