RLMF: Reinforcement Learning with Metacognitive Feedback Teaches LLMs to Know What They Don't Know
> arXiv:2606.32032 | Yale University / Google Research > Authors: Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan > Submitted to COLM 2026
---
The Core Problem
Large language models often cannot distinguish between what they know and what they don't. They rarely say "I'm not sure about this" or "I may have gotten this part wrong." The gap between what a model *says* about its confidence and what it *actually believes internally* is a fundamental alignment problem. This paper leverages metacognition—the ability to accurately assess one's own performance—as a reinforcement learning signal to close that gap.
Factual Calibration ≠ Faithful Calibration
- Factual calibration: Does the model's stated confidence match its actual accuracy? A model that says "90% confident" and is right 90% of the time is factually calibrated.
- Faithful calibration: Does the stated confidence match the model's internal beliefs? If the model internally only has 60% confidence but reports "90% confident," faithful calibration fails—even if it happens to be correct.
- Correct answers with accurate self-assessment are strongly reinforced.
- Correct answers with *inaccurate* self-assessment are down-weighted.
- Incorrect answers with *accurate* self-assessment are still reinforced—knowing you were wrong is more valuable than blind confidence.
- Standard RL → RLMF: 63% improvement
- 96% average human evaluation win rate (diversity, naturalness, helpfulness, contextual fit)
- SOTA across 10 tasks, 6 domains, with training only on PopQA and generalizing out-of-distribution to MATH, SimpleQA, etc.
- Small RLMF-trained models beat frontier models on faithful calibration: outperforming GPT-5 by 37%, Gemini-3.1-Pro by 17%, and Gemini-3-Flash by 25%
- Liu, G. K.-M., Caciularu, A., Yona, G., Szpektor, I., & Cohan, A. (2026). Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs. arXiv:2606.32032. https://arxiv.org/abs/2606.32032
- Code: https://github.com/yale-nlp/RLMF
> Paper quote: "A model may appear factually calibrated yet remain misaligned with its internal beliefs."
Internal confidence is estimated via sampling consistency: sampling the same question multiple times and checking whether answers agree.
The RLMF Method
RLMF builds on GRPO (Group Relative Policy Optimization, as used in DeepSeek-R1):
1. Self-assessment: For each generated answer, the model predicts how faithful its confidence report was (F_pred, scored 0-1).
2. Gold faithful calibration: Compare reported confidence against sampling-consistency-based internal confidence to compute F_gold.
3. Metacognitive accuracy: Z_g = 1 - (F_pred - F_gold)² — perfect self-knowledge yields the maximum reward of 1.
4. Advantage scaling: A_g^RLMF = Z_g × A_g
This dual reward means:
Metacognitive Data Selection
Rather than random or active-learning sampling, RLMF selects examples where the model thinks it performed well and thinks it performed poorly. These extremes provide complementary signals: one corrects overconfidence, the other corrects excessive self-doubt. This outperforms selecting merely "hard" examples.
Two-Stage Decoupled Framework
Stage 1 — Numerical Faithful Calibration: Train the model to output 0-1 confidence scores that faithfully reflect internal beliefs, measured by the paper's new cMFG* metric (a bias-corrected Mean Faithful Gap).
Stage 2 — Linguistic Uncertainty Mapping: Map calibrated scores to natural language expressions via targeted rewriting, e.g.:
| Score | Expression | |-------|-----------| | 0.92+ | "almost certain" | | 0.87 | "highly likely" | | 0.68 | "probable" | | 0.51 | "about even" | | 0.13 | "little chance" | | 0.07 | "almost no chance" |
Decoupling allows the linguistic mapping to be adapted per user or domain (e.g., conservative phrasing for medical users) without retraining the RL policy.
Key Results
| Method | Avg cMFG* | Improvement | |--------|-----------|-------------| | FUT (SOTA baseline) | ~0.65 | — | | SFT | ~0.64 | — | | RLMF (Stage 1) | ≥0.82-0.84 | +29% over FUT | | RLMF (two-stage) | ≥0.82-0.84 | +25% over SFT |
Reliability diagrams show that RLMF fixes the biggest weakness of raw models and FUT: poor faithful calibration at low internal confidence, where models habitually project certainty. RLMM's gains are largest exactly where models "don't know what they don't know."
Why Small Models Win
1. Frontier models are over-optimized toward confident, fluent answers, which undermines uncertainty expression. 2. RLMF is counter-intuitive training: it requires the model to say "I'm unsure"—against a large model's "nature." 3. Smaller models are easier to re-program: 7B-parameter models adapt their behavior more readily through RLMF than 70B+ models.
Limitations
1. Compute cost: each answer requires an extra self-assessment pass; cost is not quantified in the paper. 2. Imperfect internal confidence estimation: sampling consistency is only an approximation of internal beliefs. 3. Granularity: a single per-answer F_pred may miss sentence-level variation in confidence. 4. Linguistic mapping is English-centric and may need rebuilding for other languages and cultures.
The Bigger Picture
Optimizing faithful calibration can trade off against factual calibration, so RLMF includes task correctness and factual calibration terms in its reward to balance all three. More broadly, metacognition is a bridge toward self-improvement and alignment: a model that knows its weaknesses can request more context, refuse out-of-scope tasks, recognize its own error patterns, and adapt dynamically in multi-turn dialogue.
> One-line takeaway: Models can learn not only from *what they got right*, but from *knowing what they got right*—and that honesty about limitations may be the starting point of alignment.
---
References