Sampling Beats Self-Reflection: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Budgets

Overview

For the past two years, "let the model reflect on itself" has been a default pattern in Agentic AI: Self-Refine, Reflexion, Multi-Agent Debate, Best-of-N with Self-Verification — all share the same idea: generate more text, then pick the best output.

This paper does something simple that nobody had rigorously done before: compare methods again with token budgets strictly held equal.

The result is uncomfortable: across 36 paired comparisons, no self-reflection method stably beats "repeated sampling + majority vote," a naive baseline, in any configuration. 10 comparisons show self-reflection stably worse, and all 18 self-verification comparisons are negative.

Experimental Design

• 7 methods: Chain-of-Thought (baseline), Repeated Sampling + Majority Vote, Self-Refine, Reflexion, Best-of-N with Model Selection, Best-of-N with Majority Vote, Multi-Agent Debate
• 3 parameter scales: 1.5B, 3B, 7B
• 2 math benchmarks: GSM8K, MATH
• 150 problems per benchmark
• Every token counts toward cost — including tokens for critique, reflection, debate rounds, and verification
• Paired design: each method vs. equal-cost repeated sampling on the same problems, with bootstrap confidence intervals and multiple-comparison correction

Key Numbers

• 36 paired comparisons: 0 show self-reflection methods stably winning
• 10 comparisons show self-reflection stably worse — all involving methods where the model checks its own output
• 18/18 self-verification comparisons are negative
• Best-of-N: "let the model pick" vs. "just take majority vote":
• At 1.5B: model selection is 8.0 and 11.3 percentage points below majority voting
• At 7B: 2.0 and 1.3 points below (no longer significant)
• Self-Refine and forced Reflexion: still 3.6 to 10.1 points below baseline even at 7B
• The darkest-humor finding: Reflexion on the smallest model never triggered its own retry mechanism — it judged itself correct every time and silently degraded into single-shot Chain-of-Thought. The paper's own words: "It judged itself correct every time and silently became a single chain of thought."

What This Means

The gains attributed to "self-reflection" may never have come from reflection — they came from generating more samples.

When a paper claims "Self-Refine improves over CoT by 12 points," the real source of improvement is likely that Self-Refine generated 3-5x more tokens while single-shot CoT generated once. Flatten the token budget, and the "reflection" magic disappears.

More damning still: self-verification (having the model check its own output) is not just useless but harmful — the more the model checks, the more errors it introduces. This echoes "Looping Is Not Reliability" (correctness is not an absorbing state; 16% of correct patches get reverted in a second round): self-verification doesn't correct errors; it converts correct answers into wrong ones.

The scale effect is also interesting: the smaller the model, the more harmful "let the model pick" becomes; at larger scale the gap shrinks to insignificance. This suggests self-verification ability improves slowly with scale, but even at 7B it hasn't become a positive gain.

A Methodological Warning

This paper sounds an alarm for the entire Agentic AI field: many papers claiming "agents beat baselines" may be comparing against straw men — baselines with uncontrolled costs.

The correct standard: any paper claiming "method X works" must answer one question — "At equal token budget, does method X still beat repeated sampling?" If it cannot, X's effectiveness is unproven.

This connects perfectly with the "evaluation blind spot" principle: you optimize what you measure, and what you don't measure is where the problems hide. Nobody previously measured equal-cost baselines, so "self-reflection works" became the hiding place.

---

Paper: Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B Author: Iliya Mirzaei arXiv: https://arxiv.org/abs/2607.28576

FAQ

Q1: Who should read this?

Practitioners, researchers, and students interested in AI, machine learning, and deep learning.

Q2: What are the core takeaways?

• An uncomfortable conclusion for the Agent community: self-reflection never stably beats repeated sampling at equal token cost
• A rigorous experimental design across 7 methods, 3 model scales, and 2 math benchmarks
• Key numbers: 0/36 stable wins for self-reflection, 18/18 negative self-verification results

See the links in the original article.