Summary
This post summarizes arXiv paper 2506.18491 by Gina Wong, Drew Prinster, and Suchi Saria, which studies calibration in mixture-of-experts (MoE) models under distribution shift. Calibration aligns a model's predictive uncertainty with the empirical frequencies of its outcomes, which is essential for trusting reported probabilities. While prior work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration—especially in MoE models—the conditions under which calibration benefits MoE remain poorly understood. The authors analyze how routing mechanisms interact with expert-level calibration under distribution shift. Their key finding: for hard-routed models, expert-level calibration is sufficient to guarantee overall model calibration under a broad class of distribution shifts, but this does not hold for soft-routed models. To address this gap, they propose an adversarial reweighting scheme that penalizes calibration error of the routing-aggregated model under distribution shifts. Empirically, the method improves the accuracy-calibration trade-off on average and hard data subsets, across model classes, prediction tasks, and types of distribution shift. Original post sourced from zhichai.net.
Overview
Field: Machine Learning
Authors: Gina Wong, Drew Prinster, Suchi Saria
Published: 2025-06-23
arXiv: 2506.18491
Summary
Calibration aligns a model's predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration, with mixture-of-experts (MoE) models showing strong empirical improvements in particular; however, the conditions under which calibration helps MoE are not well understood.
In this work, the authors study how MoE models behave under distribution shift, focusing on how routing mechanisms interact with expert-level calibration.
Key Findings
- Hard-routed models: Expert calibration is sufficient to ensure calibration of the overall model under a broad class of distribution shifts.
- Soft-routed models: Expert-level calibration alone is insufficient to guarantee overall calibration.
- Proposed solution: An adversarial reweighting scheme that penalizes calibration error of the routing-aggregated model under distribution shift.
- Empirical results: The method improves the accuracy-calibration trade-off on average and hard data subsets, across model classes, prediction tasks, and distribution shifts.
Reference
- Paper: Toward Calibrated Mixture-of-Experts Under Distribution Shift (arXiv:2506.18491)
---
*Auto-collected on 2026-06-23*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178208036