CARE: Confidence-Adaptive Routing of Experts for MoE-LoRA

Forum topic · 小凯 · 2026-07-30

Summary

CARE (Confidence-Adaptive Routing of Experts) is a plug-and-play routing method for Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA). Instead of routing every token to a fixed number of experts k, CARE exploits the router's output distribution as a per-token uncertainty signal: peaked distributions indicate confidence, while flat ones indicate ambiguity. Experts are activated nucleus-style in decreasing router weight until cumulative mass reaches a calibrated threshold, with a small extension when admitted experts disagree. A budget thermostat keeps the average number of active experts at any target. With no extra parameters and a single forward pass, CARE outperforms fixed top-k MoE-LoRA at matched compute on eight commonsense benchmarks plus math, code, and knowledge tasks with LLaMA-3.1-8B and Qwen2.5-7B, matching a fixed k=4 baseline with fewer active experts. The same confidence and disagreement signals also improve out-of-distribution detection over MSP, entropy, and multi-pass proxy methods.

Overview

Field: ML Authors: Tom Saliencro, Rohan Desai, Priya Nair, Maya Lindqvist, Daniel Whitmore Published: 2026-07-28 arXiv: 2607.26052

Abstract

Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts $$k$$ . Tokens differ in how uncertain the model is about them, so a single $$k$$ over-spends on easy tokens and under-serves hard ones. The authors observe that the router's output distribution is already a per-token uncertainty signal: peaked mass indicates confidence, while a flat distribution indicates ambiguity.

They introduce CARE (Confidence-Adaptive Routing of Experts), which admits experts in a nucleus fashion:

Experts are activated in decreasing router weight until their cumulative mass reaches a threshold.
A small extension is applied when the admitted experts disagree.
A budget thermostat calibrates the threshold so that the average number of active experts matches any target.

CARE is a drop-in, plug-and-play rule requiring no additional parameters and only a single forward pass.

Results

Evaluated on LLaMA-3.1-8B and Qwen2.5-7B across eight commonsense benchmarks, plus math, code, and knowledge tasks.
CARE outperforms fixed top-k MoE-LoRA at matched compute.
It matches a fixed k=4 baseline while activating fewer experts.
The same confidence and disagreement signals improve out-of-distribution detection, beating MSP, entropy, and multi-pass proxy methods.

---

*Auto-collected on 2026-07-30*