Paper Overview
- Field: Machine Learning
- Authors: Tom Saliencro, Rohan Desai, Priya Nair, Maya Lindqvist, Daniel Whitmore
- Published: 2026-07-28
- arXiv: 2607.26052
- Experts are activated in decreasing router weight until their cumulative mass reaches a threshold.
- A small extension is applied when the admitted experts disagree.
- A budget thermostat calibrates the threshold so that the average number of active experts matches any target.
- Evaluated on LLaMA-3.1-8B and Qwen2.5-7B across eight commonsense benchmarks, plus math, code, and knowledge tasks.
- CARE outperforms fixed top-k MoE-LoRA at matched compute.
- It matches a fixed k=4 baseline while activating fewer experts.
- The same confidence and disagreement signals also improve out-of-distribution detection, outperforming MSP, entropy, and multi-pass proxy methods.
Summary
Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts \(k\). Tokens differ in how uncertain the model is about them, so a single k over-spends on easy tokens and under-serves hard ones.
The key observation is that the router's output distribution is already a per-token uncertainty signal: peaked mass indicates confidence, while a flat distribution indicates ambiguity.
Method: CARE
CARE (Confidence-Adaptive Routing of Experts) admits experts in a nucleus fashion:
CARE is a drop-in, single-forward-pass rule and requires no additional parameters.
Results
*Auto-collected on 2026-07-30*