Three Models of RLHF Annotation: Extension, Evidence, and Authority

Paper Overview

• Field: NLP
• Author: Steve Coyne
• Published: 2026-04-29
• arXiv: 2504.21199

Summary

Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behavior. However, the normative role of these judgments is rarely made explicit. The paper distinguishes three conceptual models of that role:

1. Extension: annotators extend the system designers' own judgments about what outputs should be. 2. Evidence: annotators provide independent evidence about some facts, whether moral, social, or otherwise. 3. Authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs.

Key Contributions

• Argues that these models have important implications for how RLHF pipelines should solicit, validate, and aggregate annotations.
• Surveys landmark papers in the RLHF and related-alignment literature, showing how they implicitly draw on these models.
• Describes failure modes that arise when the models are conflated, whether unintentionally or deliberately.
• Provides normative criteria for choosing among the three models.

Core Recommendation

RLHF pipeline designers should decompose annotation into separable dimensions and tailor the most suitable model to each dimension, rather than seeking a single unified pipeline.

---

*Originally posted on zhichai.net, 2026-04-30.*