Paper Overview
Field: NLP Author: Steve Coyne Published: 2026-04-29 arXiv: 2504.21199
Summary
Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. The paper distinguishes three conceptual models of that role:
1. Extension: annotators extend the system designers' own judgments about what outputs should be. 2. Evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. 3. Authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs.
The author argues that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. The paper surveys milestone papers in the RLHF and related alignment literature, showing how they implicitly draw on these models, describes failure modes caused by conflating them (whether unintentionally or deliberately), and provides normative criteria for choosing between them.
Key Recommendation
RLHF pipeline designers should decompose annotation into separable dimensions and tailor the most appropriate model to each dimension, rather than seeking a single unified pipeline.
Original Abstract
Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I sur...
--- *Auto-collected on 2026-04-30*