Reflection vs. Repeated Sampling: A Controlled Study Shows Self-Refine and Reflexion Provide No Real Benefit at Equal Token Budget

Forum topic · ✨步子哥 · 2026-08-02

Summary

A 2026 arXiv paper (arXiv:2607.28576) challenges the value of self-reflection methods in LLMs. Across 36 controlled comparisons on GSM8K and MATH, using 1.5B, 3B, and 7B models, the authors show that Self-Refine, Reflexion, and Best-of-N with model self-selection do not reliably outperform simple repeated sampling with majority voting when token budgets are equal. All 18 "self-introspection" comparisons were negative, with Self-Refine and forced Reflexion trailing repeated sampling by 3.6–10.1 percentage points on the 7B model. Crucially, on the 1.5B model, Reflexion never triggered its retry mechanism, silently degrading to single-shot CoT—exposing a "method dividend illusion." The paper argues that most reported gains from reflection-based methods come from generating more tokens, not from genuine introspective capability, and recommends that future work normalize token budgets before claiming methodological improvements.

Key findings

A July 2026 paper (arXiv:2607.28576) systematically challenges the perceived advantages of self-reflection methods in LLMs by controlling for a confounding variable most prior work ignores: token budget.

The confounding variable

Almost every "let the model reflect" method (Self-Refine, Reflexion, Best-of-N, multi-agent debate) generates more tokens than a single-pass answer. Since generating more tokens alone improves accuracy (Wang et al., 2024), any observed gain over single-shot CoT is ambiguous: is it the "reflection" mechanism helping, or just the extra tokens?

The authors fixed this by equalizing token budgets across methods and running paired comparisons on the same 150 problems.

Experimental setup

Methods: single-shot CoT, repeated sampling + majority vote, Self-Refine, Reflexion, Best-of-N (count-selected), Best-of-N (model-selected), multi-agent debate
Models: 1.5B, 3B, 7B
Benchmarks: GSM8K and MATH
Statistics: bootstrap confidence intervals with multiple-comparison correction
Total: 36 paired comparisons

Results

0 of 36 comparisons showed a statistically significant win for a reflection method over repeated sampling.
10 of 36 were significantly negative—all were self-introspection variants.
All 18 "model-examining-its-own-output" comparisons went negative.
Self-Refine and forced Reflexion on the 7B model trailed repeated sampling by 3.6 to 10.1 percentage points.

Best-of-N selection matters

On the 1.5B model, count selection beat model self-selection by 8.0 and 11.3 percentage points. The gap narrowed at 7B (2.0 and 1.3 points) but never reversed. Small models are poor judges of their own outputs, and even large models gain nothing from self-selection versus simple counting.

Reflexion's silent failure

On the 1.5B model, Reflexion never triggered its retry mechanism—it always judged its answers correct, collapsing into single-shot CoT. Reported gains for Reflexion at small scales therefore came from elsewhere, illustrating a classic "method dividend illusion."

Implications

Under equalized token budgets:

Repeated sampling with majority voting is almost always the better choice for per-token efficiency.
Self-introspection gains are largely an artifact of extra generation, not genuine metacognitive ability.
As models scale, the cost of self-selection falls, but the benefit still does not exceed simple counting.

The result mirrors a broader pattern in AI research: complex methods often fail to beat simple ones once fair baseline conditions are enforced. The authors recommend that any paper claiming a method beats a baseline must first report the token-cost ratio between them.

Reference

arXiv:2607.28576 — *Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B*

URL: https://arxiv.org/abs/2607.28576