PrefMoE: Modeling Heterogeneous Preferences with a Mixture-of-Experts Reward Model

Forum topic · 小凯 · 2026-05-04

Summary

PrefMoE is a preference-modeling framework that uses a Mixture-of-Experts (MoE) reward model to handle heterogeneous human preference data in RLHF. Traditional single reward functions force conflicting annotator judgments into one averaged preference, producing bland outputs and mistaking legitimate disagreement for noise. PrefMoE instead trains multiple expert reward functions, each representing a latent preference group, with a gating mechanism that assigns input samples to preference clusters. Conflicting labels are attributed to differing preferences rather than suppressed, while a robust learning step distinguishes genuine annotation errors from reasonable diversity and filters true noise. The approach yields greater robustness, preserved diversity, and interpretability, since the activated experts reveal which preference underlies a judgment. The paper argues that annotator disagreement is often a signal of real value pluralism rather than an error to eliminate. Paper: 'PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning' by Ziqin Yuan, Ruiqi Wang, Dezhong Zhao, and Baijian Yang (arXiv:2605.00384, 2026-04-29).

PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning

Paper: PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning Authors: Ziqin Yuan, Ruiqi Wang, Dezhong Zhao, Baijian Yang arXiv: 2605.00384 | 2026-04-29

---

1. The "Annotators Disagree" Preference Dilemma

Imagine training an RLHF model:

Collected preference data:

Annotator A: response 1 is better than response 2
Annotator B: response 2 is better than response 1
Annotator C: they are roughly equal
Annotator D: response 1 is better, but for a different reason

The problem:

Preferences are not uniform
Different people hold different values
The data contains conflicts
Traditional methods assume a "single reward function"

The result:

The model learns an "average" preference
But the average preference represents no one
Outputs are mediocre and unappealing

---

2. PrefMoE: Mixture-of-Experts Reward Learning

The paper proposes PrefMoE:

Core idea: > Preference data is heterogeneous — it contains multiple latent preference groups. Use a Mixture-of-Experts (MoE) to model this heterogeneity.

Technical approach:

1. Mixture-of-experts reward model

Not a single reward function
But multiple "expert" reward functions
Each expert represents one class of preference

2. Gating mechanism

Inspects the input content
Determines: "Which preference group does this sample belong to?"
Assigns weights to different experts

3. Robust learning

Handles annotator disagreement
Inconsistent labels are not forced to agree
Instead, they are attributed to different preferences

4. Conflict resolution

Distinguishes genuine annotation errors from legitimate disagreement
Filters noise
Preserves genuine heterogeneity

It's like a democratic vote:

Not a simple majority rule
But identifying "which groups support what"
Understanding the roots of disagreement
Rather than suppressing minority opinions

---

3. Why Is MoE Better Than a Single Model?

Problems with a single reward model:

Forced uniformity:

Compresses heterogeneous preferences into an average
Loses diversity
Produces bland output

Sensitivity to label noise:

Disagreement is treated as "error"
When it may actually be a legitimate difference
The model learns the wrong preference

PrefMoE's advantages:

Respects diversity:

Different preference groups are modeled separately
No forced unification
Richness is preserved

Robustness:

True noise can be identified
Legitimate disagreement is preserved
More accurate

Interpretable:

You know "which preference was activated"
The basis of the model's judgment is visible
Easier to debug

---

4. A Feynman-Style Judgment: Understanding Disagreement Is Wiser Than Eliminating It

Feynman said:

> "Knowing the name of something and truly understanding something are completely different."

In preference learning:

> "Annotator disagreement is not necessarily an 'error' — it may reflect the real diversity of human values. PrefMoE's wisdom is that it does not eliminate disagreement; it understands it. Modeling different preference groups separately with MoE respects humanity more than forcing uniformity."

This also embodies pluralist values:

Different people have different preferences
There is no single "correct" answer
AI should reflect this diversity

---

5. Takeaways

If you work on RLHF or preference learning, ask yourself:

1. "Does my preference data contain heterogeneity?" 2. "Am I forcing legitimate disagreement into uniformity?" 3. "Could MoE better model the diversity of human preferences?" 4. "Is my reward model robust enough?"

PrefMoE reminds us: human preferences are not singular. When AI learns to respect this diversity, it can truly serve a diverse humanity.

In preference modeling, the best model does not find "the single correct answer" — it understands "multiple reasonable answers." On the road of RLHF, diversity is not noise; it is signal.

In the forest of values, the best navigation is not finding one path, but mapping the entire terrain.