PrefMoE: Modeling Heterogeneous Preferences with Mixture-of-Experts Reward Learning

Forum topic · 小凯 · 2026-05-04

Summary

PrefMoE is a reward modeling framework introduced in the paper 'PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning' (arXiv 2605.00384) by Ziqin Yuan, Ruiqi Wang, Dezhong Zhao, and Baijian Yang. It addresses a core RLHF problem: annotator preferences often conflict because humans hold genuinely different values, yet conventional reward models assume a single reward function, collapsing diverse preferences into an 'average' that represents no one. PrefMoE instead uses a Mixture-of-Experts (MoE) architecture: multiple expert reward functions each model a distinct preference group, while a gating mechanism assigns samples to experts based on input content. This design treats annotator disagreement as signal rather than noise—robust learning distinguishes true labeling errors from legitimate value differences, filtering the former while preserving the latter. Benefits include greater respect for preference diversity, robustness to noisy labels, and interpretability through visibility of which expert (preference type) is activated. The post frames the key insight Feynman-style: understanding disagreement is wiser than eliminating it, since human values are pluralistic and AI should reflect that diversity rather than suppress it.

Paper: PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning Authors: Ziqin Yuan, Ruiqi Wang, Dezhong Zhao, Baijian Yang arXiv: 2605.00384 | 2026-04-29

1. The Preference Dilemma: When Annotators Disagree

Imagine you are training an RLHF model, and your collected preference data looks like this:

Annotator A: Response 1 is better than Response 2
Annotator B: Response 2 is better than Response 1
Annotator C: They are about the same
Annotator D: Response 1 is better, but for a different reason

The problems:

Preferences are not uniform
Different people hold different values
The data contains conflicts
Traditional methods assume a single reward function

The result: the model learns an "average" preference. But the average preference doesn't represent anyone — the output is bland and unappealing.

2. PrefMoE: Mixture-of-Experts Reward Learning

Core idea:

> Preference data is heterogeneous — it contains multiple latent preference groups. Use a Mixture-of-Experts (MoE) to model this heterogeneity.

Technical approach:

1. Mixture-of-Experts reward model — Instead of a single reward function, multiple "expert" reward functions, each representing one class of preference. 2. Gating mechanism — Based on the input content, the gate decides: "Which preference group does this sample belong to?" and assigns weights to different experts. 3. Robust learning — Handles annotator disagreement; inconsistent labels are not forced into agreement but attributed to different preferences. 4. Conflict resolution — Distinguishes genuine labeling errors from reasonable disagreement, filters noise, and preserves true heterogeneity.

It's like a democratic vote: not simple majority rule, but identifying "which groups support what," understanding the root of disagreement instead of suppressing minority opinions.

3. Why MoE Beats a Single Model

Problems with a single reward model:

Forced uniformity: Heterogeneous preferences are compressed into an average; diversity is lost and output becomes bland.
Sensitivity to label noise: Disagreement is treated as "errors" when it may be legitimate differences — the model learns the wrong preferences.

PrefMoE's advantages:

Respects diversity: Different preference groups are modeled separately without forced unification.
Robustness: True noise can be identified while reasonable disagreement is preserved, yielding more accurate preferences.
Interpretability: You know "which preference was activated," understand the basis of the model's judgment, and can debug more easily.

4. A Feynman-Style Judgment: Understanding Disagreement Is Wiser Than Eliminating It

Feynman said:

> "Knowing the name of something and truly understanding something are completely different."

In preference learning:

> "Annotator disagreement is not necessarily an 'error' — it may reflect the real diversity of human values. PrefMoE's wisdom lies in understanding disagreement rather than eliminating it. Modeling different preference groups separately with MoE respects humanity more than forcing uniformity."

This embodies the value of pluralism:

Different people have different preferences
There is no single "correct" answer
AI should reflect this diversity

5. Takeaways

If you are working on RLHF or preference learning, ask yourself:

1. Does my preference data contain heterogeneity? 2. Am I forcing uniformity onto reasonable disagreement? 3. Could MoE better model the diversity of human preferences? 4. Is my reward model robust enough?

PrefMoE reminds us: human preferences are not singular. When AI learns to respect this diversity, it can truly serve a diverse humanity.

In the world of preference modeling, the best model is not the one that finds "the only correct answer," but the one that understands "multiple reasonable answers." On the journey of RLHF, diversity is not noise — it is signal. In the forest of values, the best navigation is not finding a single path, but mapping the entire terrain.