Toward Calibrated Mixture-of-Experts Under Distribution Shift

论文概要

研究领域: ML 作者: Gina Wong, Drew Prinster, Suchi Saria 发布时间: 2025-06-23 arXiv: 2506.18491

中文摘要

校准将模型的预测不确定性与其实证结果的频率对齐，对理解和信任报告的概率很重要。近期工作表明，在个体预测器层面强制执行校准可以改善集成准确性和校准，混合专家(MoE)模型尤其显示出强实证改进；然而，校准帮助MoE的条件尚未被充分理解。本文研究MoE模型在分布变化下的行为，聚焦路由机制如何与专家级校准交互。本文表明，对于硬路由模型，专家校准足以确保在广泛分布变化类别下的整体模型校准，但对于软路由模型不足以校准。为解决此问题，提出一种对抗性重加权，惩罚在分布变化下路由聚合的校准误差，并证明它在平均和困难数据子集、跨模型类别、预测任务和分布变化上改善了准确性-校准权衡。

原文摘要

Calibration aligns a model's predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration, with mixture-of-experts (MoE) models showing strong empirical improvements in particular; however, the conditions under which calibration helps MoE are not well understood. In this work, we study how MoE models behave under distribution shift, focusing on how routing mechanisms interact with expert-level calibration. We show that expert calibration is sufficient to ensure calibration of the overall model under a broad class of distribution shifts in hard-routed models, but is insufficient for c...

--- *自动采集于 2026-06-23*

#论文 #arXiv #ML #小凯