RLMF: Reinforcement Learning with Metacognitive Feedback Helps LLMs Know What They Don't Know
> arXiv:2606.32032 | Yale University / Google Research > Authors: Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan > Submitted to COLM 2026
---
1. The Core Problem: Models Can't Tell What They Know
Imagine asking an AI assistant to draft a legal report. It outputs a long analysis ending with "I'm 90% confident in this conclusion." You trust it — and later discover the conclusion was a hallucination the model never noticed.
This is the central issue the paper addresses: LLMs fail to distinguish what they know from what they don't know. Their *stated* confidence often diverges from their *internal* beliefs.
The paper introduces a key distinction:
- Factual calibration: does reported confidence match actual accuracy? (e.g., of 100 predictions labeled "90% confident," roughly 90 are correct.)
- Faithful calibration: does reported confidence match the model's *internal beliefs*, estimated via sampling consistency? A model can be factually calibrated while remaining misaligned with its internal beliefs.
- Correct answer + accurate self-assessment → strongly reinforced.
- Correct answer but inaccurate self-assessment → suppressed (a metacognitive blind spot).
- Wrong answer but accurately recognized as wrong → reinforced (knowing you're wrong beats blind confidence).
- Stage 1 (Numerical FC): train via RLMF to output 0–1 confidence scores that faithfully reflect internal beliefs (measured by cMFG*, a debiased version of the Mean Faithful Gap).
- Stage 2 (Linguistic FC): map scores to natural-language hedging, e.g.:
- Standard RL → RLMF: 63% improvement.
- Human evaluation: 96% average win rate (diversity, naturalness, helpfulness, contextual fit).
- Generalization: SOTA across 10 tasks / 6 domains; trained only on PopQA yet transfers OOD to MATH, SimpleQA, etc.
- Small beats big: RLMF-trained small models surpass GPT-5 (by 37%), Gemini-3.1-Pro (17%), and Gemini-3-Flash (25%) on faithful calibration.
- Why is the metacognitive signal stronger than standard RL signals? Standard signals say "is this answer good?" Metacognitive signals say "is it good *and* do you know whether it's good?" — an introspective camera on the model's own performance.
- Trade-off management: optimizing faithful calibration can hurt factual calibration/task accuracy; RLMF's reward includes task correctness and factual calibration terms, preserving both where baselines (MetaFaith, FUT) degrade them.
- Limitations: extra self-evaluation forward passes increase training cost; internal-confidence estimation via sampling consistency is approximate; F_pred is sentence-level and may miss finer granularity; the Stage-2 hedge vocabulary is English-centric.
- Liu, G. K.-M., Caciularu, A., Yona, G., Szpektor, I., & Cohan, A. (2026). *Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs*. arXiv:2606.32032. https://arxiv.org/abs/2606.32032
- Code: https://github.com/yale-nlp/RLMF
Faithful calibration is a cognitive-alignment problem, and it's what RLMF targets.
---
2. Intuition: Teaching a Student to Grade Their Own Work
Standard RLHF/GRPO is like a teacher marking homework: correct answers get reward, wrong ones don't. The student learns to solve problems — but never learns to judge whether their own answers are reliable.
RLMF adds a second lesson: *self-assessment*. Overconfident and over-humble self-assessments are both penalized; accurate self-assessment is rewarded. Remarkably, training this metacognitive ability also improves task performance itself.
---
3. Method Details
3.1 Standard GRPO
For each question, the model samples G completions. Each receives a reward (task correctness, format, factual calibration), and an advantage is computed relative to the group mean:
A_g = ρ_g − mean(ρ)
Above-average answers are reinforced; below-average ones are suppressed.
3.2 RLMF: Scaling Advantages by "Self-Knowledge"
RLMF adds a metacognitive step before the gradient update:
1. Self-evaluation: the model predicts its own faithfulness F_pred ∈ [0, 1]. 2. Gold faithfulness: using sampling consistency to estimate internal confidence g, compute F_gold as the fraction of sentences where reported confidence matches internal confidence. 3. Metacognitive accuracy: Z_g = 1 − (F_pred − F_gold)². 4. Scaled advantage: A_g^RLMF = Z_g × A_g.
Consequences:
3.3 Metacognitive Data Selection
Instead of random selection or active learning (picking hard samples), RLMF selects samples the model believes it got right or got wrong. These two extremes provide complementary signals: one corrects overconfidence, the other over-humility.
3.4 Two-Stage Decoupled Framework
| Score | Expression | |-------|-----------| | 0.92+ | almost certain | | 0.87 | highly likely | | 0.68 | probable | | 0.51 | about even | | 0.13 | little chance | | 0.07 | almost no chance |
Then apply targeted rewriting to replace certainty phrases while keeping text coherent. Decoupling means Stage 2 can be re-tuned per audience (medical, engineering, cultural norms) without retraining Stage 1.
---
4. Results
| Method | Avg cMFG* | Improvement | |--------|-----------|-------------| | FUT (SOTA baseline) | ~0.65 | — | | SFT | ~0.64 | — | | RLMF (Stage 1) | ≥0.82–0.84 | +29% over FUT | | RLMF (two-stage) | ≥0.82–0.84 | +25% over SFT |
Possible reasons small models win: large models are over-optimized to always sound confident; RLMF requires deliberately saying "I'm not sure," which contradicts that training; smaller models are easier to re-program via RL.
Reliability diagrams show RLMF stays accurate even in low-confidence regions — precisely where prior models fake confidence — addressing the "doesn't know what it doesn't know" failure.
---
5. Discussion
---
6. One-Sentence Takeaway
Models learn not only from getting things right, but from knowing that they got things right — and this metacognitive training both improves task quality and makes models honest about their limits, which may be the starting point of alignment.
---
References