The Question: Reflect or Just Sample Again?
When a language model answers a math problem incorrectly, you have two options:
A. Make it reflect — "You got that wrong. Think about why, and try again." B. Make it try again — Same question, new random seed, take the majority answer.
Intuitively, A seems smarter. But a July 2026 paper shows that at equal token budgets, B almost always beats A, by a significant margin.
Paper: *Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B* (arXiv:2607.28576)
Method Gains vs. Budget Gains
A commonly overlooked confounder: almost all "reflection" methods make the model generate more tokens.
- Self-Refine: answer, critique, revise — three rounds of output
- Reflexion: answer, reflect, retry — multiple rounds
- Best-of-N: N times the output
- Debate: multiple model instances — even more
- 7 methods: single-pass CoT, repeated sampling (majority vote), Self-Refine, Reflexion, Best-of-N (model-selected), Best-of-N (count-selected), debate
- 3 model scales: 1.5B, 3B, 7B
- 2 math benchmarks: GSM8K and MATH
- 150 problems, paired comparisons
- Rigorous statistics: bootstrap confidence intervals + multiple-comparison correction
- Count-selected: generate 8 answers, pick the most frequent
- Model-selected: generate 8 answers, let the model pick the best
- At unconstrained token budgets, Self-Refine and Reflexion can still improve accuracy
- But if you care about per-token efficiency (as you should in production), repeated sampling + majority voting is almost always better
- The "intelligence" of model self-review may be a method-gain illusion — the benefit comes from simply generating more tokens
Generating more tokens by itself improves accuracy. Wang et al. (2024) showed that sampling a question multiple times and taking the majority beats a single pass. This means if a "reflection method" beats a single answer, you can't tell whether reflection is working or whether it's just the extra tokens.
The paper does something simple that no one had done: flatten the token budget — give reflection methods and repeated sampling the same budget, then compare accuracy.
36 Controlled Comparisons, Zero Wins
Experimental setup:
The one-line result: no self-review method reliably beats repeated sampling at equal token budget. Across 36 comparisons: 0 significantly positive, 10 significantly negative (all self-review methods), 26 indistinguishable from zero.
More striking: all 18 self-review comparisons were negative or null. Self-Refine and forced Reflexion scored 3.6 to 10.1 percentage points below repeated sampling on 7B models.
The Cost of "Choosing"
Best-of-N has two selection modes:
Intuition says model selection should win. But on 1.5B models, count-selection beat model-selection by 8.0 and 11.3 percentage points. At 7B, the gap shrank to 2.0 and 1.3 points — no longer significant.
The lesson: having a model review its own output is an inherently costly operation. Tokens spent judging, comparing, and selecting would earn more if used for additional samples.
Reflexion's Embarrassment
Reflexion is designed to judge whether its answer is correct and retry if wrong. But on 1.5B models, Reflexion never triggered its retry mechanism — it always judged itself correct, silently degrading into single-pass CoT. The "Reflexion effect" reported in the original paper, on small models, wasn't reflection at all.
This is a classic method-gain illusion: the method looks like it's doing something, but the improvement comes from elsewhere.
What This Means
A Deeper Principle
When a method claims to beat a baseline, first ask how many extra tokens it costs.
A method that uses 3× tokens for a 5% accuracy gain shouldn't be published if a simple baseline gets 6% at the same cost. Academia has long neglected this budget-flattened control — partly because "letting models reflect" is a sexy narrative, while "sample more and take the majority" sounds too plain.
The paper's value isn't overturning a few methods — it's doing a rigorous version of a control experiment everyone skipped.
---
Paper link: https://arxiv.org/abs/2607.28576