Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost

Forum topic · ✨步子哥 · 2026-08-03

Summary

A 2026 paper (arXiv:2607.28576) challenges the assumed value of self-reflection methods in language models. Across 36 controlled comparisons on GSM8K and MATH benchmarks using 1.5B, 3B, and 7B models, the authors found that repeated sampling with majority voting consistently outperforms Self-Refine, Reflexion, and Best-of-N variants when token budgets are equalized. Zero comparisons showed significant gains for self-reflection methods, while 10 showed significant losses, especially for models forced to critique their own outputs. Self-Refine and Reflexion underperformed repeated sampling by 3.6 to 10.1 percentage points at the 7B scale. The study also reveals that asking small models to select their best answer is worse than simple counting, and Reflexion on 1.5B models never triggered its retry mechanism, silently degrading to single-shot CoT. The findings suggest that apparent benefits of reflection methods are largely a "budget illusion"—extra generated tokens rather than genuine reasoning improvements.

Background

When a language model answers a math problem incorrectly, two recovery strategies are common:

A. Reflection — ask the model to critique its output and try again (Self-Refine, Reflexion).
B. Repeated sampling — sample the same prompt multiple times with different random seeds and take a majority vote (Best-of-N with counting).

Intuition favors A, but a 2026 paper titled *Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B* (arXiv:2607.28576) runs 36 controlled comparisons and finds the opposite.

Method Bonus vs. Budget Bonus

Self-Refine produces three rounds of output. Reflexion adds reflection and retry rounds. Best-of-N generates N answers. Debate multiplies inference across instances. All of these naturally consume more tokens than a single-chain-of-thought response.

Crucially, generating more tokens already improves accuracy. Wang et al. (2024) showed that sampling a question multiple times and taking a majority beats a single sample. This creates a confound: when a "reflection" method beats single-shot inference, it is unclear whether the gain comes from reflection or simply from extra tokens.

This paper's core contribution is budget equalization: both reflection methods and repeated sampling are given the same token budget before comparison.

36 Comparisons, Zero Wins

Experimental setup:

7 methods: single-shot CoT, repeated sampling (majority vote), Self-Refine, Reflexion, Best-of-N (model selection), Best-of-N (counting), debate.
3 model scales: 1.5B, 3B, 7B.
2 benchmarks: GSM8K and MATH.
150 paired questions per comparison.
Statistics: bootstrap confidence intervals with multiple-comparison correction.

Headline result: no self-reflection method reliably beat repeated sampling at equal token cost. Of 36 comparisons, 0 were significantly positive for reflection, 10 were significantly negative (all involving self-critique), and 26 were indistinguishable from zero. All 18 comparisons involving self-critique were negative. Self-Refine and forced Reflexion underperformed repeated sampling by 3.6 to 10.1 percentage points at 7B.

The Cost of "Selection"

Best-of-N can aggregate outputs two ways:

Counting: choose the most frequent answer.
Model selection: have the model pick its own best answer.

Counting beat model selection by 8.0 and 11.3 percentage points at 1.5B. The gap shrank to 2.0 and 1.3 points at 7B, where it became non-significant. Small models are poor judges of their own outputs; larger models are slightly better, but still do not beat simple counting.

Reflexion's Embarrassment

On 1.5B models, Reflexion never triggered its retry mechanism — it judged itself correct every time and silently degenerated to single-shot CoT. The reported "Reflexion effect" on small models is therefore a methodological illusion: the method never actually ran, but appeared to perform differently.

Implications

For production systems where token efficiency matters, repeated sampling plus majority voting is almost always the better default.
For research, the apparent intelligence of self-reflection may be a budget illusion masquerading as a method bonus.
General principle: when a method claims to beat a baseline, first ask how many extra tokens it spent. A 5% gain at 3× cost is not a contribution if a simple 3× baseline achieves 6%.

The paper does not claim reflection is useless in unconstrained-budget settings — Self-Refine and Reflexion do improve absolute accuracy there. But when token efficiency is the metric, the conclusion is stark.

Paper Reference

arXiv:2607.28576 — *Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B*.

https://arxiv.org/abs/2607.28576