pass@k Can Deceive You: EliteReplay Uses an Evolutionary Archive to Fight Solution Collapse in RL Post-Training

Forum topic · 小凯 · 2026-10-17

Summary

EliteReplay, a method from Ohio State University, addresses 'behavior sharpening' in RL post-training of language models, where GRPO boosts pass@1 but silently erases most of the base model's diverse correct solution strategies. The paper argues that pass@k is a 'scalar illusion': it rewards any correct output without capturing how many distinct strategies the model retains. EliteReplay works in two steps: first, it builds a quality-diversity archive via MAP-Elites using the frozen base model, covering multiple solution categories (e.g., JOIN vs. subquery vs. window function in SQL); second, it trains with GRPO plus a 5% replay loss on archived solutions. On BIRD text-to-SQL, it matches baselines on pass@k while nearly doubling unique-correct-strategy U@128 (9.1 vs. 6.2) and retaining 62% of base-model strategies versus ~39% for baselines. It also doubles ambiguity-coverage on AmbiQT and unexpectedly improves OOD math performance. Ablations show both the QD archive and continuous replay are essential. Code and models are open-sourced.

pass@k Can Deceive You: How EliteReplay Fights Solution Collapse in RL Post-Training with an Evolutionary Archive

A Counterintuitive Finding

Imagine training a language model to write SQL. Before training, the model can write five different correct queries for the same question — some with JOINs, some with subqueries, some with window functions. After one round of GRPO post-training, pass@1 jumps from 25% to 55%. Great.

Then you sample 128 outputs and find the model now uses essentially one style.

What happened to the other four? They're gone. Not "optimized," but starved — RL concentrates probability mass on the highest-reward path while suppressing other correct paths. This is behavior sharpening: RL doesn't create new capabilities, it amplifies a few behaviors the base model already has.

The question becomes: what metric do you use to measure diversity loss?

If you use pass@k (the probability of at least one correct answer in k samples), things may look fine — pass@128 is still 73%. But pass@k only asks "is there a correct one," not "how many distinct correct ones are there." A model can score high pass@k by repeatedly outputting the same correct solution while forgetting all the others the base model knew.

This is the core insight of the EliteReplay paper: pass@k is a scalar illusion that compresses the high-dimensional quantity of solution diversity into one bit.

The Paper: EliteReplay's Two-Step Approach

EliteReplay comes from Hanane Nour Moussa, Doeun Lee, and Sachin Kumar at Ohio State University, in a paper titled *Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search*.

Step 1: Build an "Elite Archive" with MAP-Elites

Before training, run Quality-Diversity (QD) search on a 1000-question subset of the training set using the frozen base model, via the classic MAP-Elites algorithm:

1. Define the behavior space: rather than hand-designing descriptors, an LLM labeler inspects seed solutions and induces "solution categories" (e.g., "JOIN-based," "subquery-based," "window-function"), combined with structural features (SQL AST features). 2. Initialize the archive: insert correct base-model samples into their archive cells. 3. Select targets: prefer solution categories not yet present in the archive. 4. Mutate: prompt the base model to generate new solutions conditioned on a target category and a parent elite. 5. Evaluate and insert: verify correctness; insert into empty cells, or replace existing elites if the new solution is shorter.

When the search budget is exhausted, each question has an archive of correct SQL solutions covering multiple categories, one elite per category.

Step 2: Joint GRPO + Replay Training

During training, the objective becomes:

\mathcal{L}(\theta) = (1-\mu) \mathcal{L}_{\text{GRPO}}(\theta) + \mu \mathcal{L}_{\text{replay}}(\theta)

where the replay loss is the length-normalized negative log-likelihood of archived solutions, with $\mu = 0.05$ — 95% of gradient from GRPO, 5% from archive replay.

Crucially, the archive is built offline with the frozen base model — not collected dynamically during training or from RL rollouts. Once built, it is fixed; training only replays from it.

Why pass@k Is a Scalar Illusion

The paper proposes three levels of diversity metrics:

1. pass@k: probability of at least one correct answer in k samples. Coarsest — checks correctness, not strategy count. 2. U@k (unique correct strategies): the number of distinct strategy categories among correct solutions in k samples, computed deterministically from SQL AST features without an LLM judge. 3. Effective diversity: the probability that two random samples are both correct and use different strategies — penalizes both inaccuracy and lack of diversity.

Plus strategy retention: how many of the base model's correct strategies survive training.

Results:

| Method | BIRD pass@1 | BIRD pass@128 | BIRD U@128 | Strategy retention | |--------|------------|---------------|------------|--------------------| | Base model | 25.3% | 73.6% | — | 100% | | GRPO | ~55% | 68–69% | ~3.0 | 34–39% | | MaxRL | ~55% | 73.7% | 6.2 | ~39% | | EliteReplay | 52.5% | 74.8% | 9.1 | 62% |

MaxRL and EliteReplay have nearly identical pass@128 (73.7% vs. 74.8%) but U@128 differs by 50% (6.2 vs. 9.1). If you only look at pass@k, they seem equivalent; but MaxRL relies on repeatedly outputting a few strategies, while EliteReplay preserves most of the base model's solution spectrum.

Disambiguation: Diversity in Practice

The paper uses the AmbiQT benchmark, where every question admits two valid interpretations (e.g., "department employees" could mean current staff or everyone ever assigned). A model with only one interpretation can never cover the other, no matter how many times you sample.

At k=128, GRPO covers both interpretations on only 4.2% of questions; the strongest diversity baseline MaxRL reaches 14.2%; EliteReplay reaches 27.0% — nearly double MaxRL and closing more than half the gap to the base model's 37%.

When questions are inherently ambiguous, solution diversity is coverage.

Ablations: Both Conditions Are Necessary

Q1: Does the archive's composition matter, or is replay alone enough?

Replacing the MAP-Elites archive with IID-sampled correct solutions keeps pass@k similar but drops U@128 from 9.1 to 7.4. IID sampling over-represents common solutions; rare strategies still get suppressed by RL. The archive's broad-coverage structure is necessary.

Q2: Is SFT-before-training enough, or must replay be continuous?

An "Elite SFT → GRPO" variant (SFT on the archive, then plain GRPO without replay) drops to ordinary GRPO levels (68.5% pass@128, 3.0 U@128). SFT warm-up cannot prevent collapse — RL re-suppresses the rare strategies it learned. Replay must run throughout RL training.

Design principle: diversity preservation = QD archive (broad coverage) + continuous replay (constant injection).

An Unexpected Bonus: OOD Generalization

On MATH-500 (math tasks never seen in training), EliteReplay raises pass@1 from the base model's 28.3% to 35.5%, while all three baselines drop to 25–27%. The authors speculate replay acts as "rehearsal," mitigating out-of-domain forgetting — suggesting QD archives may also protect general capabilities against RL's typical narrowing.

Engineering Notes

Archive construction: one-time cost, fixed search budget on 1000 questions with the frozen base model.
Training: GRPO + 5% replay, 100 steps. Replay loss is pure supervised learning with no extra rollout cost.
Evaluation: U@k needs 128 samples, much more expensive than pass@1, but necessary for measuring diversity.

Code: https://github.com/skai-research/EliteReplay

Includes the archive builder (strategy labeling, mutation, SQL validation, MAP-Elites insertion), released BIRD archives (2708 occupied cells, 780 questions, 1500 replay samples), training code for EliteReplay and all baselines, BIRD/Spider evaluation code, diversity metric implementations, and the released model hananour/EliteReplay-BIRD-4B.

Reflections: Three Levels of Insight

1. The scalar illusion in evaluation. The paper's sharpest contribution is its deconstruction of pass@k, the de facto standard in RL post-training reports (DeepSeek-R1, Qwen3, Llama-4). If two methods share pass@k but one relies on 1 solution and the other on 9, is their "reasoning ability" really equivalent? Every time we find a scalar illusion, we should ask: what high-dimensional information did this scalar compress away?

2. The nature of RL forgetting. The ablations reveal something counterintuitive: RL forgets what SFT learned — not "never learned" but "learned then overwritten." RL gradients systematically suppress non-optimal paths even when they are correct. EliteReplay's replay is an anti-forgetting device that anchors solutions destined to be forgotten.

3. QD and RL are complementary. RL aims to converge (find the optimal policy); QD aims to diverge (cover the behavior space). EliteReplay separates the goals: QD discovers offline (with the base model), RL optimizes online (GRPO), and replay bridges them. This "offline archive + online replay" pattern generalizes to any setting needing diversity during optimization — agent policy training, generative sample diversity, exploration-exploitation balance.

Limitations and Outlook

The paper acknowledges two limitations: 1. Model scale: experiments only on Qwen3.5-4B-Base; larger models are untested. 2. Domain: only text-to-SQL, where SQL's structure makes strategy classification feasible. Defining "strategies" for math reasoning remains open.

Still, the method applies wherever there is verifiable correctness and meaningful behavioral distinctions — e.g., code optimization, algorithm generation.

Conclusion

EliteReplay's most exciting contribution is exposing an evaluation blind spot. When two methods share the same pass@k, one may be "one solution repeated," the other "nine solutions in full spectrum" — these are very different capabilities downstream.

Next time you read an RL post-training paper, ask: behind its pass@k, what is the U@k?

---

Paper: arxiv.org/abs/2610.22290 Code: github.com/skai-research/EliteReplay Model: hananour/EliteReplay-BIRD-4B (HuggingFace)