Sample More, Reflect Less: Self-Reflection Methods Lose to Repeated Sampling at Equal Token Cost

Forum topic · ✨步子哥 · 2026-08-01

Summary

A controlled study challenges the consensus that self-reflection methods improve LLM reasoning. Across 36 paired comparisons on GSM8K and MATH using 1.5B, 3B, and 7B models, no self-refinement method consistently outperformed simple repeated sampling with majority voting when token budgets were held equal. Worse, methods that let a model inspect its own outputs (Self-Refine, Reflexion, model-based Best-of-N selection) were reliably worse, with 18 out of 18 self-inspection comparisons going negative. Best-of-N with model selection trailed majority voting by 8.0 to 11.3 points at 1.5B and by 2.0 to 1.3 points at 7B. Self-Refine and Reflexion still trailed Chain-of-Thought by 3.6 to 10.1 points at 7B. On the smallest model, Reflexion never triggered its retry loop, judging itself correct every time and silently degrading into one-shot CoT. The results suggest that reported gains from reflection largely come from extra token spend, not from reflection itself, and that self-inspection can actively corrupt correct answers, raising methodological questions for the entire agentic AI literature.

Background

For roughly two years, "let the model reflect on itself" has been a near-default design pattern in agentic AI. Self-Refine, Reflexion, Multi-Agent Debate, and Best-of-N with Self-Verification all share one assumption: generating more text and selecting the best outcome yields better reasoning. This paper asks a simple but rarely tested question: what happens when we strictly equalize token budgets before comparing these methods to a baseline?

Experimental Setup

Methods (7 total): Chain-of-Thought (baseline), Repeated Sampling + Majority Vote, Self-Refine, Reflexion, Best-of-N with Model Selection, Best-of-N with Majority Vote, Multi-Agent Debate
Model sizes: 1.5B, 3B, 7B
Benchmarks: GSM8K and MATH, 150 problems per benchmark
Cost accounting: every token counts, including critique, reflection, debate rounds, and verification steps
Evaluation design: paired comparison against equal-cost repeated sampling, with bootstrap confidence intervals and multiple-comparison corrections

Key Numbers

36 paired comparisons, 0 wins for reflection methods under equal token cost
10 comparisons where reflection methods were significantly worse, all involving the model checking its own outputs
18/18 self-inspection comparisons were negative
Best-of-N model-selection vs. majority voting:
1.5B: -8.0 and -11.3 percentage points
7B: -2.0 and -1.3 percentage points (no longer significant)
Self-Refine and Reflexion at 7B: still 3.6 to 10.1 points below the CoT baseline
Darkest finding: Reflexion on the smallest model never triggered its retry mechanism, judged itself correct every time, and silently degraded into a single chain of thought. Paper quote: *"It judged itself correct every time and silently became a single chain of thought."*

What This Means

The gains attributed to "self-reflection" may never have come from reflection at all; they came from generating a few extra samples. When you see a paper claiming "Self-Refine improves over CoT by 12 points," the real source of the gain is likely that Self-Refine produced 3-5x more tokens than a single CoT call. Once token budgets are equalized, the reflection magic disappears.

More troubling: self-inspection is not just useless, it is harmful. The model becomes more wrong the more it checks itself. This mirrors the observation from "Looping Is Not Reliability" that 16% of correct patches get reverted in a second pass; self-inspection is not just unable to fix errors, it can rewrite correct outputs into incorrect ones.

There is also a scaling signal: the smaller the model, the more harmful "let the model pick" becomes. At 7B the gap shrinks to non-significance, suggesting self-inspection capability improves slowly with scale, but even at 7B it has not turned into a net positive.

A Methodological Warning

This paper issues a warning for the entire agentic AI field: many papers claiming "Agent X beats baseline" may be comparing against a straw man whose cost has not been controlled. Any paper that claims "method X works" should be required to answer: *Does X still beat repeated sampling at equal token cost?* If not, the effectiveness of X is unproven.

This connects to the "evaluation blind-spot law": optimize what you measure, and what you do not measure is where problems hide. Because nobody previously measured the equal-cost baseline, "self-reflection works" became one of those hiding spots.

---

Paper title: Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B Author: Iliya Mirzaei arXiv: https://arxiv.org/abs/2607.28576