RLHF (Reinforcement Learning from Human Feedback)

RLHF is the technology that enables AI to truly understand human intent. It uses reinforcement learning combined with human feedback to train large language models so their outputs align with human values.

Core Goals

• Align AI models with human values
• Through learning from human preferences, make model outputs safer, more helpful, and harmless

The 3H Principles

• Helpful
• Harmless
• Honest

Analogy: The Growth of an AI Student

1. Pre-training: Like a student learning foundational knowledge — acquiring language understanding and generation abilities. 2. SFT: Teaching the student how to read and solve problems — gaining instruction-following ability. 3. RLHF: A teacher grades homework, and the student keeps improving based on feedback.

The Three-Stage RLHF Training Pipeline

Stage 1: Supervised Fine-Tuning (SFT)

Training an obedient base model.
• Data: instruction datasets (instruction-response pairs)
• Method: supervised learning with cross-entropy loss
• Goal: teach the model to understand and follow basic instructions
• Output: an SFT model capable of following instructions

Stage 2: Reward Model (RM) Training

Training a judge that can score.
• Data: human preference data (chosen vs. rejected)
• Method: learning-to-rank, maximizing the score gap between chosen and rejected responses
• Goal: learn to evaluate text quality and understand human preferences
• Output: a reward model that can score text

Stage 3: PPO Reinforcement Learning Training

Teaching the model to earn high scores.
• Algorithm: PPO (Proximal Policy Optimization)
• Models: four models working together (Actor, Ref, Reward, Critic)
• Goal: maximize the reward given by the reward model
• Output: a final model aligned with human preferences

The PPO Training Process

1. Actor Model — executes the policy and generates responses 2. Reference Model — provides a baseline 3. Reward Model — evaluates response quality 4. Critic Model — estimates expected returns

Advantages of RLHF

1. Alignment with Human Values

Following the 3H principles ensures outputs conform to human ethics and avoids generating harmful content.

2. Better User Experience

More natural responses that better fit user needs, improving conversation quality.

3. Data Efficiency

Small amounts of high-quality feedback yield large performance gains. A 1.3B-parameter InstructGPT outperformed the 175B-parameter GPT-3.

Challenges

1. Inconsistent Human Feedback

Different annotators make subjective judgments that vary.

2. Reward Model Bias

The RM may not fully reflect true human preferences and is limited by its training data.

3. Compute Requirements

Requires substantial GPU resources and long training times; four models are trained simultaneously, with memory demands of roughly 240GB.

Future Directions

• DPO (Direct Preference Optimization): learns human preferences directly, without training an RM or running RL.
• Efficient training algorithms: lightweight architectures, distributed training, and model compression to cut costs.
• Enhanced safety: multi-layer safety checks, adversarial training, and red-team testing.

Why It Matters

RLHF is widely regarded as the most critical breakthrough separating ChatGPT from GPT-3 — the key technology on the path from GPT-3 to ChatGPT.

---

Source: Easy AI learning platform; created for AI knowledge popularization.