Paper: Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B arXiv: 2607.28576
---
An experiment that embarrasses the "reflection camp"
You have probably heard the story: let a large model grade its own work — write an answer, reflect on what's wrong, revise, and repeat. Self-Refine and Reflexion are the loudest flagships of this approach, cited thousands of times and practically the default move in "reasoning enhancement."
But this paper asks a question that should have been asked long ago: when the number of tokens spent is held equal, does "iterative reflection" actually beat "sampling several times and voting"?
The answer is no. On models from 1.5B to 7B, across six or seven benchmark tasks, repeated sampling consistently beats Self-Refine and Reflexion at an equal token budget.
Why "equal tokens" is the fair comparison
Earlier literature claimed "Self-Refine beats plain generation," hiding a huge sleight of hand: Self-Refine runs several rounds, each consuming tokens; plain generation runs only one. Comparing a method that eats five portions of tokens against one that eats one is not a fair win.
This paper controls for the variable: both approaches get the same token budget. The reflection camp spends it on "generate-critique-rewrite" loops; the sampling camp spends it on N independent samples followed by majority voting or best-of selection. The results:
- On reasoning tasks like GSM8K, GPQA, and MATH, repeated sampling wins consistently.
- The larger the model (7B vs 1.5B), the bigger sampling's advantage — since larger models produce higher-quality single samples, more samples more often hit the correct answer.
- The "improvements" from reflection often just rewrite an already-correct first answer into a more mediocre version — reflection isn't correcting errors, it's flattening them.
What is "reflection" actually doing?
The paper makes a sharp observation: much of the "improvement" in Self-Refine and Reflexion happens when the original answer was already correct — the model critiques itself, changes right answers to wrong ones, then changes them back. This back-and-forth burns enormous tokens for near-zero or negative net gain.
In other words, the core problem with reflection methods is not "can it fix errors" but that it doesn't know when to revise and when not to. It forces a "critique-rewrite" pass on every problem, and that pass is itself noisy — the model's self-critique makes mistakes just like its generation does.
Repeated sampling is fundamentally different: it doesn't require the model to "know where it went wrong," only that the model "occasionally gets it right." And the evidence shows letting the model guess several times and picking the best is more reliable than making the model find fault with itself.
Why this matters
This result is a heavy blow to the whole "reasoning enhancement" field.
Over the past two years, "making models reflect" became almost a belief — CoT, Self-Consistency, Self-Refine, Reflexion, endless methods to make models "think more steps." But this paper suggests that part of these methods' gains may come not from reflection itself but from spending several times more tokens. Once token budgets are equalized, reflection's halo fades.
This echoes earlier findings: reasoning "epiphanies" often reflect having sampled enough times rather than genuine reasoning. Sampling more is a poor-man's best-of-n, the most cost-effective reasoning enhancement.
A deeper implication
The "from 1.5B to 7B" in the title is not decoration. It shows the phenomenon is not an artifact of one model size but a scale-independent pattern. Small models, large models, same story.
This implies something uncomfortable: current LLMs' self-critique ability may be far weaker than we assumed. A model's critiques come from the same distribution as its answers, sharing the same biases and blind spots. Letting a model critique itself is like letting a student write and grade their own exam — not useless, but far less reliable than an independent source (even another independent sample).
Reflection, a core human cognitive ability, may not truly exist in current LLMs. The "improvements" we see are more a leverage effect of token budgets than a leap in cognition.
Practical advice
If you're building an LLM reasoning task on a limited budget, the paper's advice is direct:
1. Prefer repeated sampling + majority voting over Self-Refine or Reflexion.
2. At equal token budgets, sampling is simpler to implement — no critique prompts, no multi-turn dialogue management, just sample N times and take the mode.
3. If you must use reflection, first benchmark against equal-token sampling — you may find the reflection pipeline you painstakingly built underperforms a one-line for i in range(N).
Honest limitations
The paper only tests 1.5B–7B models. Would reflection methods win at 70B+ or GPT-4-class scale? Unknown. Larger models may have genuinely stronger self-critique and positive net gains from reflection — but that needs experimental proof, not assumption.
Also, the comparison is "equal tokens," not "equal time" or "equal money." Sampling is naturally parallel; reflection is naturally serial — in latency-sensitive settings, reflection may retain value.
But the core conclusion is solid: under the token budget — the most common comparison basis — reflection loses to sampling. At minimum, past papers claiming "Self-Refine substantially improves performance" owe us an equal-token controlled experiment.
---
Paper link: https://arxiv.org/abs/2607.28576
Related code: original Self-Refine repo https://github.com/madaan/self-refine (the paper itself provides no independent code repository)