Ask a large language model a factual question it doesn't know. What does it do? Most models guess — because during training, guessing correctly earns points, guessing wrong loses points, but "I don't know" scores the same as a wrong guess. From an optimization standpoint, guessing beats abstaining.
This is the abstention dilemma: any scoring rule that rewards correct answers while penalizing wrong answers and abstentions identically systematically encourages models to guess instead of admitting ignorance.
A recent paper by Juan Terven et al. ran controlled experiments with Qwen3-14B comparing two routes to teach abstention. The result is counterintuitive: the cheap route won.
Two Routes: Policy Training vs. a Filter on Top
Route A: Policy training (RL). Train the answering model itself, via reinforcement learning, to choose between answering and abstaining. Correct answers are rewarded, wrong answers penalized, abstention scores zero. The model learns on its own when to say "I don't know."
Route B: Learn an answer selector. The answering model stays unchanged and keeps generating answers. On top of it sits a small, separately trained selector that looks at the question and the model's answer and decides whether to release it; otherwise it outputs "I don't know."
Route A is "teach the model when to shut up"; Route B is "let the model talk, but add an external reviewer." Intuition favors Route A as more elegant — but the experiments say otherwise.
The Numbers
Main results on TriviaQA:
| Metric | Original calibrated filter | Learned selector | Gain | |---|---|---|---| | Coverage (fraction of answers released) | 20.2% | 30.9% | +10.7pp | | Utility | 0.0242 | 0.0425 | +0.0183 | | ROC AUC (confidence ranking quality) | 0.779 | 0.852 | +0.073 | | Brier score (calibration quality) | 0.232 | 0.197 | -0.035 |
The selector released more answers (coverage up from 20% to 31%) and the released answers were more accurate (utility up 75%). Its confidence judgments also improved: ROC AUC rose from 0.779 to 0.852, Brier score dropped from 0.232 to 0.197.
The RL route? "Inconclusive." From the paper:
> "The inconclusive RL results do not show that reinforcement learning cannot teach abstention."
Failing to even refute RL suggests the training signal is too weak.
Why Did RL Struggle?
The paper identifies a deep problem: the reward signal itself is flawed.
If all sampled responses are answers (no abstentions), then changing the confidence threshold τ subtracts the same constant from every reward. Reward differences across thresholds are zero, so the model gets no signal about when to abstain. The model only abstains occasionally once it has learned the signal — but it can't learn the signal because it never abstains. A chicken-and-egg dilemma.
The selector route sidesteps this entirely. It doesn't require the answering model to change behavior; it learns an external judgment function from the model's actual outputs ("the model gave this answer; was it correct?").
Analogy: teaching a chatterbox silence. Route A tries to train the chatterbox to know when to stop talking — but the chatterbox never stops, so there's no signal to learn from. Route B assigns an editor who learns when to cut the mic. The editor doesn't need the chatterbox to change; just to recognize unreliable answers.
Cost Comparison
RL training required: 3 training seeds, 3 thresholds × 4 responses × 3 exposures = 36 forward passes per question, roughly 8,000 training labels.
Selector training required: the same 8,000 labels (a byproduct of the RL runs), a logistic regression or small MLP, and seconds of training time.
A cheaper ablation, "TRAIN logistic," used the same labels but raw model logit differences as features: utility 0.0385, below the selector's 0.0425. The labels help, but the selector's advantage isn't just the labels — it learns signal beyond raw logits.
Transfer to NQ-Open
Trained on TriviaQA, tested on NQ-Open:
- Utility improved from -0.0308 (original filter) to -0.0120
- The gain (0.0188) is nearly identical to the TriviaQA gain (0.0183)
- But overall utility is still negative — NQ-Open is too hard for Qwen3-14B even with a selector
Engineering Lessons for LLMs
1. Decoupling beats end-to-end. Asking one model to both generate answers and judge whether to release them creates tension: the generation objective encourages answering, the judgment objective encourages caution. Splitting the goals across two components works better — echoing PTP-U's propose–project framework.
2. RL is not a panacea. Agentic RL is trendy, but when the reward signal has a structural defect (like the constant-shift problem here), RL may learn nothing useful. Check the reward structure for usable gradients before reaching for RL.
3. Cheap can beat expensive. A logistic-regression-level selector, trained in seconds, beat heavy RL sampling. For the specific task of "should this answer be released?", an external discriminator is more efficient than internal policy adjustment.
A Sobering Analogy
Imagine an exam. Student A is trained to judge whether they can solve a problem and raise a hand to say "I can't." Student B solves problems as usual, but a teacher stands by, reviewing each answer and deciding whether to submit it. Intuitively, Student A is more advanced — real metacognition. In practice, Student A can't learn "when I don't know" because they answer everything and have no reference point for ignorance. Student B's teacher needs no change from the student, only the ability to spot unreliable answers.
For teaching models to say "I don't know," assigning a teacher is currently more practical than training student metacognition. Models may eventually learn genuine metacognition — but that day hasn't arrived.
---
Paper: Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection
Code: github.com/jrterven/Can-LLMs-learn-to-abstain
Authors: Juan Terven, Diana-Margarita Córdova-Esparza, Ruben Gomez, et al.
Model: Qwen3-14B, evaluated on TriviaQA and NQ-Open