Teaching Language Models to Embrace Uncertainty: Multi-Answer RL for Distributional Reasoning

Forum topic · 小凯 · 2026-03-27

Summary

Reinforcement learning with verifiable rewards (RLVR) sharpens language models toward a single 'most likely' answer, causing mode collapse: models lose output diversity and struggle to surface multiple valid solutions. Researchers at MIT (Isha Puri et al., arXiv:2603.24844) propose Multi-Answer RL, which redefines correctness as a set: the model generates K distinct candidate answers within one reasoning chain and earns reward for each correct one. An extended variant, Multi-Answer RLCR, adds calibration rewards based on a multi-answer Brier score so each answer carries a confidence value. Evaluated on DDXPlus medical diagnosis, a modified HotPotQA with deliberately incomplete context, and MBPP coding tasks, the method improves coverage and pass@1, boosts answer diversity, and cuts token costs (3 answers in 235 tokens vs 512 for repeated single-answer sampling on MBPP, a 54% saving). RLCR Multi also significantly improves calibration (lower Brier scores and ECE) across all benchmarks. The approach internalizes inference-time Best-of-K search into a single forward pass, supporting probabilistic, decision-support-oriented AI in high-stakes domains like medicine and law.

This forum post is a Feynman-style explainer of the MIT paper "Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models" (Puri et al., arXiv:2603.24844, March 2026). Below is a structured English rendition of its content.

Key points

The problem — mode collapse: Standard RL post-training rewards a single correct answer, collapsing the model's rich output distribution into one "safest" response. Prior work (Lin et al. 2025; Yu et al. 2025; Jin et al. 2025; Wu & Choi 2025) shows RL sharply reduces entropy: pass@1 improves while pass@k degrades.
Why it matters: Ambiguous real-world tasks — differential diagnosis, underspecified QA, code generation (multiple valid implementations), scientific hypotheses, legal analysis — require listing several plausible answers, like a physician's differential diagnosis.
Limitations of inference-time fixes: Best-of-K parallel sampling and sequential exploration (Reflexion, self-guided search) suffer from train–test mismatch and cost K× more tokens.

Method: Multi-Answer RL

The reward is set-based: given gold answer set Y* and model output set A = {a1…aK}, reward is the count of outputs in Y*: R_multi(A, Y*) = Σᵢ 1[aᵢ ∈ Y*]. Each correct answer earns a point.
The framework unifies standard RLVR (N=1, K=1), Best-of-K / pass@K objectives (N=1, K>1), and partial/full coverage settings (N>1).
Multi-Answer RLVR: the model emits K distinct answers inside one chain of thought using structured tags (<answer1> … <answer3>), with a uniqueness reward preventing duplicates and a set-level reward.
Multi-Answer RLCR: adds per-answer confidence scores qᵢ ∈ [0,1] and subtracts a multi-answer Brier penalty: R = R_multi_RLVR − (1/K)·Σᵢ(qᵢ − 1[aᵢ ∈ Y*])², based on strictly proper scoring rules (Gneiting & Raftery 2007), incentivizing honest calibrated confidence.

Experiments and results

Benchmarks: DDXPlus (medical differential diagnosis, N≥1, K=3, 25k training samples), HotPotQA-Modified (multi-hop QA with passages deliberately removed to induce uncertainty), and MBPP (coding, multiple correct implementations verified via unit tests and AST distinctness).

Coverage and accuracy (K=3): On DDXPlus, coverage rises from 0.76 (single-answer RLVR) to 1.03 correct answers per question, diversity from 1.05 to 2.19, tokens drop from 1467 to 622. On MBPP, coverage 0.98 → 1.35, pass@1 0.29 → 0.49 (+69%), and 3 answers cost only 235 tokens vs 512 for repeated sampling — a 54% saving.
Diversity insight: Sampling 30 answers total, the multi-answer model produces substantially more unique correct answers than the single-answer model, showing mode collapse limits access to the correct-answer space itself.
Calibration: RLCR Multi lowers Brier score and ECE on all three datasets (e.g., DDXPlus top-1 Brier 0.42 → 0.27; MBPP 0.42 → 0.26), meaning the model knows when it may be wrong. One caveat: on HotPotQA, top-1 ECE slightly regresses, possibly due to a learned prior that confidences sum to 1.

Significance and future directions

Shifts AI from single-truth evaluation toward probabilistic thinking with quantified confidence.
Internalizes Best-of-K search into a single forward pass: O(1) instead of O(K) compute, valuable for resource-constrained deployment.
Provides a decision-support interface (ranked hypotheses with confidences) suited to high-stakes medicine and law, rather than opaque single answers.
Future work: adaptive/dynamic K, hierarchical answer sets, tool-verified hypotheses, and multimodal extensions.

References (as cited in the post)

1. Puri, I., et al. (2026). Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models. arXiv:2603.24844. 2. Guo, D., et al. (2025). DeepSeek-R1. arXiv:2501.12948. 3. Damani, M., Puri, I., et al. (2025). Beyond binary rewards. arXiv:2507.16806. 4. Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules. JASA 102(477). 5. Tchango, A. G., et al. (2022). DDXPlus dataset. 6. Yang, Z., et al. (2018). HotpotQA. EMNLP 2018. 7. Austin, J., et al. (2021). Program synthesis with large language models. arXiv:2108.07732. 8. Wu, F., & Choi, Y. (2025). The invisible leash. AI for Math Workshop @ ICML 2025. 9. Yue, Y., et al. (2025). arXiv:2504.13837. 10. Xiong, M., et al. (2024). Can LLMs express their uncertainty? ICLR 2024.