PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
Paper: PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning Authors: Ziqin Yuan, Ruiqi Wang, Dezhong Zhao, Baijian Yang arXiv: 2605.00384 | 2026-04-29
---
1. The "Annotators Disagree" Preference Dilemma
Imagine training an RLHF model:
Collected preference data:
- Annotator A: response 1 is better than response 2
- Annotator B: response 2 is better than response 1
- Annotator C: they are roughly equal
- Annotator D: response 1 is better, but for a different reason
- Preferences are not uniform
- Different people hold different values
- The data contains conflicts
- Traditional methods assume a "single reward function"
- The model learns an "average" preference
- But the average preference represents no one
- Outputs are mediocre and unappealing
- Not a single reward function
- But multiple "expert" reward functions
- Each expert represents one class of preference
- Inspects the input content
- Determines: "Which preference group does this sample belong to?"
- Assigns weights to different experts
- Handles annotator disagreement
- Inconsistent labels are not forced to agree
- Instead, they are attributed to different preferences
- Distinguishes genuine annotation errors from legitimate disagreement
- Filters noise
- Preserves genuine heterogeneity
- Not a simple majority rule
- But identifying "which groups support what"
- Understanding the roots of disagreement
- Rather than suppressing minority opinions
- Compresses heterogeneous preferences into an average
- Loses diversity
- Produces bland output
- Disagreement is treated as "error"
- When it may actually be a legitimate difference
- The model learns the wrong preference
- Different preference groups are modeled separately
- No forced unification
- Richness is preserved
- True noise can be identified
- Legitimate disagreement is preserved
- More accurate
- You know "which preference was activated"
- The basis of the model's judgment is visible
- Easier to debug
- Different people have different preferences
- There is no single "correct" answer
- AI should reflect this diversity
The problem:
The result:
---
2. PrefMoE: Mixture-of-Experts Reward Learning
The paper proposes PrefMoE:
Core idea: > Preference data is heterogeneous — it contains multiple latent preference groups. Use a Mixture-of-Experts (MoE) to model this heterogeneity.
Technical approach:
1. Mixture-of-experts reward model
2. Gating mechanism
3. Robust learning
4. Conflict resolution
It's like a democratic vote:
---
3. Why Is MoE Better Than a Single Model?
Problems with a single reward model:
Forced uniformity:
Sensitivity to label noise:
PrefMoE's advantages:
Respects diversity:
Robustness:
Interpretable:
---
4. A Feynman-Style Judgment: Understanding Disagreement Is Wiser Than Eliminating It
Feynman said:
> "Knowing the name of something and truly understanding something are completely different."
In preference learning:
> "Annotator disagreement is not necessarily an 'error' — it may reflect the real diversity of human values. PrefMoE's wisdom is that it does not eliminate disagreement; it understands it. Modeling different preference groups separately with MoE respects humanity more than forcing uniformity."
This also embodies pluralist values:
5. Takeaways
If you work on RLHF or preference learning, ask yourself:
1. "Does my preference data contain heterogeneity?" 2. "Am I forcing legitimate disagreement into uniformity?" 3. "Could MoE better model the diversity of human preferences?" 4. "Is my reward model robust enough?"
PrefMoE reminds us: human preferences are not singular. When AI learns to respect this diversity, it can truly serve a diverse humanity.
In preference modeling, the best model does not find "the single correct answer" — it understands "multiple reasonable answers." On the road of RLHF, diversity is not noise; it is signal.
In the forest of values, the best navigation is not finding one path, but mapping the entire terrain.