Multi-Answer Reinforcement Learning: Teaching Language Models to Embrace Uncertainty

Forum topic · 小凯 · 2026-03-27

Summary

A detailed summary of an MIT research paper (arXiv:2603.24844) introducing Multi-Answer Reinforcement Learning (RL) for language models. The paper addresses mode collapse caused by standard single-answer RL with verifiable rewards (RLVR), where models become overconfident and generate narrow outputs even when multiple valid answers exist. The authors propose two algorithms: Multi-Answer RLVR, which rewards a set of K distinct candidate answers, and Multi-Answer RLCR, which adds calibration rewards based on Brier scores so models output confidence values for each answer. Experiments on DDXPlus (medical differential diagnosis), a modified HotPotQA, and MBPP (code generation) show coverage gains from 0.76 to 1.03 on DDXPlus and Pass@1 improvements from 0.29 to 0.49 on MBPP, while cutting token usage by 54% compared with independent Best-of-K sampling. Calibration metrics such as Brier score and ECE also improve. The work reframes LLM training for open-ended, ambiguous tasks including medicine, law, and code synthesis, and discusses extensions to dynamic K, hierarchical answer sets, and tool use.

Overview

This article translates a MIT team's research (Puri et al., arXiv:2603.24844) on Multi-Answer Reinforcement Learning, a framework that trains language models to produce multiple plausible answers with calibrated confidence rather than collapsing onto a single output.

Key points

The mode collapse problem
Standard RL with verifiable rewards (RLVR) optimizes a single correct answer.
Recent work (Lin et al. 2025; Yu et al. 2025; Jin et al. 2025; Wu & Choi 2025) shows RLVR sharply lowers output entropy and degrades pass@k even as pass@1 improves.
The issue is harmful in domains that genuinely have multiple valid responses: differential diagnosis, ambiguous QA, code generation with several correct implementations, scientific hypothesis generation, and legal analysis.
Why inference-time workarounds are insufficient
Parallel sampling (Best-of-K) and sequential self-revision (Reflexion, self-evaluation-guided search) reduce pass@1 cost only by paying K times the inference compute.
They create a train-test mismatch: training rewards a single decisive answer, testing demands several distinct ones.
Human experts instead enumerate hypotheses inside one reasoning trace, which is what the paper aims to internalize.
Formalizing Multi-Answer RL
Replace the single gold answer y* with a set Y* = {y*_1, ..., y*_N} and the model's output with a set A = {a_1, ..., a_K}.
Set-level reward: R_multi(A, Y*) = Σ 1[a_i ∈ Y*], i.e. one point per correct guess.
The framework unifies several regimes: single-answer RLVR (N=1, K=1), Best-of-K (N=1, K>1), partial coverage (K ≤ N), and full coverage (K ≥ N).
Algorithm 1: Multi-Answer RLVR
The model generates K distinct candidates in a single reasoning chain, separated by tags such as <answer1>...</answer1>.
A uniqueness term prevents degenerate repetition.
Final reward is computed over the whole answer set.
Algorithm 2: Multi-Answer RLCR (calibration)
The model additionally emits a confidence q_i ∈ [0,1] per answer.
Reward: R_multi_RLCR = R_multi_RLVR - R_multi_Brier, where

R_multi_Brier = (1/K) Σ (q_i - 1[a_i ∈ Y*])^2

Brier scoring is a strictly proper scoring rule (Gneiting & Raftery, 2007), so the model is penalized for overconfidence.
In the N=1 case the confidences form a discrete distribution summing to ≤1; in the N>1 case they parameterize independent Bernoulli variables, drawing a connection to conformal prediction without requiring a fixed coverage guarantee.
Benchmarks and results
DDXPlus (Tchango et al., 2022), 25k patients, K=3 differential diagnoses. Coverage rises from 0.76 to 1.03, diversity from 1.05 to 2.19, and average tokens drop from 1,467 to 622.
HotPotQA-Modified (Yang et al., 2018) with deliberately removed context. Pass@1 climbs from 0.19 to 0.27 and coverage from 0.21 to 0.27.
MBPP (Austin et al., 2021) for code generation, judged with AST-based diversity. Pass@1 jumps from 0.29 to 0.49 (+69%), coverage from 0.98 to 1.35, diversity from 2.09 to 2.98, and tokens from 512 to 235 (-54%).
With the same total budget of 30 answers, Multi-Answer RL yields more unique correct outputs than 30 independent samples from a single-answer model, indicating that internal exploration recovers answers the single-answer model has forgotten.
Calibration (Brier, top-1 ECE, set ECE) improves on all three datasets when switching from RLVR Multi to RLCR Multi, e.g. MBPP Brier drops from 0.42 to 0.26.
One residual gap: on HotPotQA, RLCR Multi's top-1 ECE slightly worsens because confidences are constrained to sum to 1; set-level calibration still improves.
Why the approach matters
Shifts evaluation from a single-truth view to a probabilistic view of model output.
Cuts wall-clock and token cost by generating K candidates in one forward pass rather than K independent passes.
Produces auditable, decision-support outputs (e.g. tuberculosis 0.50, pneumonia 0.30, bronchitis 0.20) instead of opaque single answers.
Suggests a shift from AI as autonomous answerer to AI as collaborator that surfaces hypotheses for human experts.
Future directions highlighted by the authors
Adaptive K based on question difficulty.
Hierarchical answer sets where some diagnoses subsume others.
Tight integration with external verification tools, search engines, and databases.
Extension to multimodal tasks (vision, audio, video).

Key references

Puri, I., Damani, M., Shenfeld, I., Ghassemi, M., Andreas, J., & Kim, Y. (2026). *Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models.* arXiv:2603.24844.
Gneiting, T., & Raftery, A. E. (2007). *Strictly proper scoring rules, prediction, and estimation.* JASA 102(477).
Damani, M. et al. (2025). *Beyond binary rewards: Training LMs to reason about their uncertainty.* arXiv:2507.16806.
Wu, F., & Choi, Y. (2025). *The Invisible Leash: Why RLVR May Not Escape Its Origin.* AI for Math Workshop @ ICML 2025.
Guo, D. et al. (2025). *DeepSeek-R1.* arXiv:2501.12948.
Xiong, M. et al. (2024). *Can LLMs Express Their Uncertainty?* ICLR 2024.

Bottom line

Multi-Answer RL reframes language-model training so that producing a diverse, calibrated set of answers is the optimization target rather than a side effect recovered by expensive inference-time tricks. The empirical gains on medical diagnosis, ambiguous QA, and code generation suggest that "learning to be appropriately uncertain" is a tractable training objective and a useful direction for high-stakes deployment.