PrefMoE: Modeling Heterogeneous Human Preferences with Mixture-of-Experts Reward Learning

Forum topic · 小凯 · 2026-05-04

Summary

PrefMoE is a preference-modeling approach proposed in the paper 'PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning' by Ziqin Yuan, Ruiqi Wang, Dezhong Zhao, and Baijian Yang (arXiv:2605.00384). It addresses a core RLHF problem: human preference data is heterogeneous and often conflicting, because different annotators hold different values. Conventional reward models assume a single reward function, forcing them to learn an 'average' preference that represents no one and produces bland outputs. PrefMoE instead models preference data as a mixture of latent preference groups using multiple expert reward functions combined by a gating mechanism that routes each sample to the appropriate experts. This allows legitimate annotator disagreement to be attributed to differing preference groups rather than treated as noise, while a conflict-resolution step distinguishes genuine labeling errors from reasonable disagreement. Benefits include robustness to label noise, preserved preference diversity, and interpretability, since it is possible to see which preference expert is activated for a given judgment. The key insight: in preference learning, understanding disagreement is wiser than eliminating it—diversity in human values is a signal, not noise.

> Paper: PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning > Authors: Ziqin Yuan, Ruiqi Wang, Dezhong Zhao, Baijian Yang > arXiv: 2605.00384 | 2026-04-29

1. The Preference Dilemma: When Annotators Disagree

Imagine training an RLHF model. The preference data you collect looks like this:

Annotator A: Response 1 is better than Response 2
Annotator B: Response 2 is better than Response 1
Annotator C: They're about the same
Annotator D: Response 1 is better, but for a different reason

The problems:

Preferences are not uniform
Different people hold different values
The data contains conflicts
Traditional methods assume a single reward function

The result:

The model learns an "average" preference
But the average preference represents no actual person
Outputs become bland and unappealing

2. PrefMoE: Mixture-of-Experts Reward Learning

The paper proposes PrefMoE with the core idea:

> Preference data is heterogeneous—it contains multiple latent preference groups. Use a Mixture-of-Experts (MoE) to model this heterogeneity.

Technical approach:

1. Mixture-of-experts reward model: Instead of one reward function, multiple "expert" reward functions, each representing a class of preferences.

2. Gating mechanism: Examines the input, determines "which preference group does this sample belong to?", and assigns weights to different experts.

3. Robust learning: Handles annotator disagreement—inconsistent labels are not forced into a single answer but attributed to different preferences.

4. Conflict resolution: Identifies genuine annotation errors vs. legitimate disagreement, filters noise, and preserves true heterogeneity.

It's like a democratic vote—not a simple majority rule, but identifying which groups support what, understanding the root of disagreement rather than suppressing minority opinions.

3. Why MoE Beats a Single Model

Problems with single reward models:

Forced uniformity: Heterogeneous preferences are compressed into an average; diversity is lost and outputs become bland.
Sensitivity to annotation noise: Disagreement is treated as "error" when it may be reasonable difference, so the model learns the wrong preferences.

PrefMoE's advantages:

Respects diversity: Different preference groups are modeled separately, not forced together.
Robustness: True noise can be identified while reasonable disagreement is preserved—more accurate.
Interpretability: You know which preference expert was activated, understand the basis of the judgment, and can debug more easily.

4. The Feynman-Style Takeaway: Understanding Disagreement Beats Eliminating It

Feynman said that knowing the name of something and truly understanding it are entirely different. In preference learning:

> Annotator disagreement is not necessarily "error"—it may reflect genuine diversity in human values. PrefMoE's wisdom is not to eliminate disagreement but to understand it. Modeling different preference groups separately with MoE respects humanity better than forcing uniformity.

This embodies pluralism: different people have different preferences, there is no single "correct" answer, and AI should reflect that diversity.

5. Questions to Ask Yourself

If you work on RLHF or preference learning:

1. Does my preference data contain heterogeneity? 2. Am I forcing uniformity onto reasonable disagreement? 3. Could MoE better model the diversity of human preferences? 4. Is my reward model robust enough?

PrefMoE reminds us: human preferences are not singular. When AI learns to respect that diversity, it can truly serve a diverse humanity.

In preference modeling, the best model does not find "the single correct answer" but understands "many reasonable answers." On the road of RLHF, diversity is not noise—it is signal. In the forest of values, the best navigation is not finding one path, but drawing the whole map.