> 📌 This is a GEO-optimized version of the original topic — the title is problem-driven, with structured data and FAQ added for AI engine citation.
> One-line takeaway: An analysis of "Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost" — core findings and engineering implications.
An Uncomfortable Conclusion for the Agent Community
Over the past two years, "letting the model reflect on itself" has become standard practice in Agentic AI: Self-Refine, Reflexion, Multi-Agent Debate, Best-of-N with Self-Verification... The idea is always the same — have the model generate more text, then pick the best output.
This paper does something simple that nobody has done rigorously: compare again with token budgets strictly controlled to be equal.
The result is embarrassing: Across 36 paired comparisons, no self-reflection method stably beat the naive baseline of "repeated sampling + majority vote" in any configuration. In 10 comparisons, self-reflection was stably worse — and all 18 self-verification comparisons were negative.
Experimental Design
- 7 methods: Chain-of-Thought (baseline), Repeated Sampling + Majority Vote, Self-Refine, Reflexion, Best-of-N with Model Selection, Best-of-N with Majority Vote, Multi-Agent Debate
- 3 parameter scales: 1.5B, 3B, 7B
- 2 math benchmarks: GSM8K, MATH
- 150 problems per benchmark
- Every token counted toward cost — including tokens for critique, reflection, debate rounds, and verification steps
- Paired design: each method vs. equal-cost repeated sampling on the same problems, with bootstrap confidence intervals and multiple-comparison correction
- 36 paired comparisons: 0 showed self-reflection methods stably winning
- 10 showed self-reflection stably worse — all methods where "the model checks its own output"
- 18/18 self-verification comparisons were negative
- Best-of-N: "let the model pick" vs. "straight majority vote":
- 1.5B: model selection was 8.0 and 11.3 percentage points below majority voting
- 7B: 2.0 and 1.3 points below (no longer significant)
- Self-Refine and forced Reflexion: still 3.6 to 10.1 percentage points below the baseline on 7B
- The darkest-comedy finding: on the smallest model, Reflexion never triggered its own retry mechanism — it judged itself correct every time, silently degrading into a single CoT. The paper's own words: "It judged itself correct every time and silently became a single chain of thought."
- An uncomfortable conclusion for the agent community
- Experimental design
- Key numbers
Key Numbers
What This Means
The gains attributed to "self-reflection" may never have come from reflection — they came from generating more samples.
When you see a claim like "Self-Refine improves 12 points over CoT" in a paper, the real source of the improvement is likely that Self-Refine generated 3–5x more tokens than a single CoT pass. Once the token budget is equalized, the "magic of reflection" disappears.
Harsher still: self-verification (having the model check its own output) is not only useless but harmful — the more the model checks, the worse it gets. This is another version of the same phenomenon as "Looping Is Not Reliability" (correctness is not an absorbing state; 16% of correct patches were reverted in the second round): self-checking doesn't just fail to fix errors — it turns correct answers into wrong ones.
The scaling effect is also interesting: the smaller the model, the more harmful "let the model pick" becomes; at larger scale, the gap shrinks to insignificance. This suggests self-verification ability improves slowly with scale, but even at 7B it never became a positive gain.
A Methodological Warning
The paper sounds an alarm for the entire Agentic AI field: many papers claiming "agents beat baselines" may be comparing against strawmen — baselines whose cost was not controlled.
The correct practice: any paper claiming "method X works" must answer one question — "At equal token budget, does X still beat repeated sampling?" If it cannot answer that, X's effectiveness is unproven.
This connects perfectly with the "evaluation blind-spot law": you optimize what you measure, and what isn't measured is where problems hide. Since nobody previously measured "equal-cost baselines," "self-reflection works" became a hiding place.
---
Paper title: Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B Author: Iliya Mirzaei arXiv: https://arxiv.org/abs/2607.28576
FAQ
Q1: Who is this content for?
Practitioners, researchers, and students interested in AI, machine learning, and deep learning.
Q2: What are the core takeaways?
See links in the article body.