Efficient Exploration at Scale: DeepMind Pushes RLHF Data Efficiency Up to 1000x
Reinforcement Learning from Human Feedback (RLHF) is undergoing a quiet revolution.
Google DeepMind's latest paper, *Efficient Exploration at Scale* (arXiv:2603.17378), proposes an online reinforcement learning algorithm that achieves, with fewer than 20K human labels, what traditional offline RLHF needs 200K labels for—a 10x efficiency gain. A more aggressive extrapolation: 1M online labels can match 1B offline labels, i.e. 1000x.
This is not a victory of parameters. It is a victory of how you ask questions.
---
The Bottleneck of Traditional RLHF: You're Asking the Wrong Questions
The offline RLHF pipeline is fixed: 1. Sample a batch of responses from a baseline model 2. Have humans label which response is better 3. Train a reward model, then optimize the policy with PPO/REINFORCE 4. The model gets stronger, but the training data still comes from the old model
The problem: as the model evolves, old data no longer covers its blind spots. Like asking a middle schooler to grade college entrance exams—they aren't unwilling to answer; they simply don't know where to look.
DeepMind sees this clearly. They argue the core of RLHF should be Reinforcement Learning, and the essence of RL is exploration.
---
Three Innovations Working in Synergy
The paper's core idea is Information-Directed Exploration, but it does not work in isolation. Three components interlock, and none can be removed.
1. Affirmative Nudge: Preventing Online Learning's Cliff
Online RLHF has a notorious failure mode: *tanking*—a sudden collapse in performance after a certain number of training batches. Prior work could only lower the learning rate or roll back to old checkpoints to delay the crash.
DeepMind's fix is minimal: add a small positive constant ε to every reinforcement signal.
Mathematically, the original policy-gradient signal p(Y⪰Y'|X) - 1/2 becomes p(Y⪰Y'|X) - 1/2 + ε.
The intuition: do not give negative feedback to "roughly good" responses. In online learning, the model changes at every step; yesterday's "correct answer" may only be "okay" today. If the penalty is too strong, the model overcorrects, swings between extremes, and eventually falls off the cliff.
The comparison is直观 (see Figure 4): without the nudge, online RLHF collapses within dozens of rounds; with it, performance keeps rising.
2. Epistemic Neural Network: The Reward Model No Longer "Pretends to Be Certain"
A traditional reward model outputs a scalar: "this response scores 7.3." But it never tells you how confident it is.
DeepMind adds an uncertainty-aware component to the reward model:
- Prior Network: fixed, provides a baseline prediction
- Differential Networks: 100 independent MLP heads, each giving a slightly different score for the same response
- Final reward = Prior + average(Differentials)
- Offline RLHF: needs 200K labels to reach roughly 65% win rate
- Information-directed exploration: reaches the same level with under 20K labels
- Extrapolation: 1M labels can match 1B offline labels—1000x efficiency
- Offline RLHF's answer: a long, tangled derivation ending in "33.33 km"—not among the options.
- Information-directed exploration's answer: concise, clear, and correct: 50 km.
- Asghari et al. (2026). *Efficient Exploration at Scale*. arXiv:2603.17378. Google DeepMind.
The variance across the 100 heads for a given response pair is the model's uncertainty about that pair.
The cost is tiny: on a 9B-parameter model, the extra component is a negligible fraction.
3. Information-Directed Exploration: Spending Human Labels Where They Matter
With uncertainty measured, the next step is choosing the questions worth asking.
For each prompt, the model samples 16 responses, enumerates all candidate comparison pairs, and computes the uncertainty variance for each pair. The pair with the largest variance is submitted for human labeling.
Intuition: if the model is already certain about A vs. B, the human label merely confirms known information—low value. If it is unsure about C vs. D, the human's choice is a high-information signal that shrinks the model's epistemic blind spots.
This is essentially active learning—but not traditional uncertainty sampling. It is information gain maximization: directing a limited annotation budget at the most instructive cases.
---
Results: The Data-Efficiency Scaling Law Is Rewritten
Experiments run on Gemma 9B, with an SFT top-1 policy baseline. Human feedback is simulated with a reward model trained by Gemini 1.5 Pro (far larger than Gemma 9B, simulating a "smarter-than-the-model" annotator).
Core findings from Figure 1:
A concrete example (a math problem from the paper):
The model did not get "smarter"—the training process used human feedback more efficiently, so the model learned faster what a good answer looks like.
---
Why 1000x? The Scaling-Law View
Figure 9 of the paper performs extrapolation analysis, fitting w(n) = 1 - 0.5(n/a)^(-b).
Offline RLHF scales slowly: 10x more data yields only marginal gains, because it always trains the new model on data from the old model—the larger the distribution shift, the lower the efficiency.
Online exploration scales steeply: the model evolves at every step while question selection becomes ever more precise. Data utilization and model capability grow together, creating a compound-interest effect.
DeepMind's conclusion is blunt: 1000x is not a ceiling—just the starting point verifiable by current experiments.
---
Limitations and Outlook
The paper lists several future directions:
1. Prompt selection: currently only response pairs are optimized for a given prompt; next, choose which prompts are worth asking 2. Multi-turn dialogue: requires value models to predict long-term returns 3. Agent scenarios: exploration must be redesigned under delayed-action feedback 4. AI-assisted feedback: as comparing complex responses gets harder for humans, an AI could first argue "why A beats B," and humans just verify
---
One-Sentence Summary
DeepMind turned RLHF from "collect data, then batch-process" into "learn while asking—asking the most valuable questions": the affirmative nudge stabilizes online training, epistemic neural networks quantify uncertainty, and information-directed exploration decides what to ask each round.
10x verified; 1000x within sight.
---
Reference: