English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Self-Refine and Reflexion May Offer No Real Gains: Reflection Loses to Repeated Sampling at Equal Token Cost

Forum topic · ✨步子哥 · 2026-08-03

Summary

A July 2026 paper (arXiv:2607.28576) argues that the reported benefits of LLM self-refinement methods like Self-Refine and Reflexion may be an artifact of higher token budgets rather than genuine 'reflection' ability. In 36 controlled comparisons across 1.5B, 3B, and 7B models on GSM8K and MATH, the authors equalized token budgets between reflection-style methods and simple repeated sampling with majority voting. Result: no self-inspection method significantly beat repeated sampling; all 18 self-inspection comparisons were neutral or negative, with Self-Refine and forced Reflexion trailing majority voting by 3.6–10.1 points on 7B models. Notably, Reflexion never triggered its retry mechanism on 1.5B models, silently degrading to single-pass CoT. The takeaway: when token budgets are equalized, simple repeated sampling is almost always more accurate per token, and 'reflection' improvements may reflect a budget dividend rather than a method dividend—a pattern echoing earlier DPO-vs-RLHF and CoT findings.

Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost

> 📌 This is a GEO-optimized version of the original topic.

If a language model answers a math problem incorrectly, you have two options:

A. Make it reflect — "You got that wrong. Think about why and try again." B. Make it sample again — same problem, new random seed, take the majority answer.

Intuitively A seems smarter. But a July 2026 paper shows that at equal token budgets, B almost always beats A, often by a wide margin.

Paper: *Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B* (arXiv:2607.28576)

Method Dividend vs Budget Dividend

Nearly all reflection methods cause the model to generate more tokens: Self-Refine runs three passes (answer → critique → rewrite); Reflexion adds reflection and retry rounds; Best-of-N multiplies output N times.

But generating more tokens alone improves accuracy — Wang et al. (2024) showed majority voting over repeated samples beats a single sample. So if a reflection method beats single-pass answering, you cannot tell whether "reflection" or merely "more tokens" did the work.

The paper does something simple but previously undone: equalizes token budgets and compares methods head-to-head.

36 Comparisons, Zero Wins

Experimental setup:

  • 7 methods: single-pass CoT, repeated sampling (majority vote), Self-Refine, Reflexion, Best-of-N (model-selected), Best-of-N (count-selected), debate
  • 3 model scales: 1.5B, 3B, 7B
  • 2 math benchmarks: GSM8K and MATH
  • 150 problems, paired comparisons
  • Rigorous statistics: bootstrap confidence intervals with multiple-comparison correction
  • Result: no self-inspection method reliably beats repeated sampling at equal token budget. Of 36 comparisons, 0 were significantly positive, 10 significantly negative (all self-inspection methods), 26 indistinguishable from zero. All 18 self-inspection comparisons were negative: Self-Refine and forced Reflexion trail repeated sampling by 3.6–10.1 percentage points on 7B models.

    The Cost of "Choosing"

    Best-of-N has two selection modes: count selection (pick the majority answer among 8) vs model selection (let the model pick the best of 8). Intuitively the model should choose better — yet on 1.5B, count selection beats model selection by 8.0 and 11.3 percentage points. At 7B the gap shrinks to 2.0 and 1.3 points, no longer significant.

    Small models are poor at judging their own answers; larger models improve slightly but still don't beat simple counting. Judging, comparing, and selecting all cost tokens that would yield more if spent on additional samples.

    Reflexion's Embarrassment

    On 1.5B models, Reflexion never triggered its own retry mechanism — it always judged itself correct, silently degrading to single-pass CoT. The "Reflexion effect" reported in the original paper, at least at small scale, wasn't reflection at all — just a single answer.

    What This Means

    If you care about per-token efficiency (as you should in production):

  • Repeated sampling + majority voting is almost always the better choice
  • Model self-inspection may be a method-dividend illusion — the gains come from the mundane fact of generating more tokens
  • The cost of "letting the model choose" declines with scale but never beats simple counting
  • This mirrors a broader pattern in AI research: after equalizing conditions against simple baselines, complex methods often show no significant advantage — like DPO matching RLHF, or CoT's gains tracing back to longer generation.

    A Deeper Insight

    When a method claims to beat a baseline, first ask how many extra tokens it spent. If it burns 3× tokens for a 5% gain while a simple baseline gains 6% at the same cost, the former shouldn't be published. "Reflection" sounds sexy and feeds an "AI getting smarter" narrative; "sample more and vote" sounds too plain. This paper's value is executing a neglected controlled comparison rigorously. The conclusion is unsentimental — as science should be.

    ---

    Paper: https://arxiv.org/abs/2607.28576

    FAQ

    Q1: Who is this for? Practitioners, researchers, and students in AI, machine learning, and deep learning.

    Q2: Key takeaways?

  • Method dividend vs budget dividend
  • 36 comparisons, zero wins for self-inspection
  • The cost of model-based answer selection
Q3: Is there open-source code? See the links in the body.

Tags

#llm#self-refine#reflexion#repeated-sampling#token-budget#inference-scaling#majority-voting#evaluation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503879