Reasoning with Sampling: Your Base Model is Smarter Than You Think

A training-free Power Sampling algorithm from a Harvard University research team (published on arXiv, 16 Oct 2025) claims base language models are far better reasoners than standard sampling suggests — and that inference-time sampling can rival reinforcement-learning (RL) fine-tuning.

Key points

• Core question: Does RL fine-tuning teach models new knowledge, or merely sharpen their existing output distributions toward correct answers (the "distribution sharpening" hypothesis)?
• Proposal: Sample directly from a *power distribution* of the base model:

P_power(x) ∝ [P_base(x)]^α, with α > 1

This amplifies high-probability (good) answers without any training.

• Algorithm: Since exact sampling is intractable, the authors use an MCMC (Metropolis–Hastings) procedure that generates text block by block and iteratively refines each block.

Why not low-temperature sampling?

At step t, the two approaches differ in where the exponent α sits:

• Low-temperature style (the impatient writer):

propensity ∝ ( Σ_{x_{t+1:T}} P(x_{t:T}|x_{1:t-1}) )^α

Sum first, then exaggerate — one brilliant future path can be drowned out by many mediocre ones.

• Power sampling (the strategic writer):

propensity ∝ Σ_{x_{t+1:T}} [P(x_{t:T}|x_{1:t-1})]^α

Exaggerate each future path first, then sum — a single excellent route dominates the decision.

Implementation: block-wise Metropolis–Hastings

Hyperparameters: exponent α (e.g., 4.0), max length L (e.g., 512), block size B (e.g., 32), MCMC steps K (e.g., 10) per stage.

1. Autoregressively generate the first B-token draft. 2. For K rounds: pick a random position i, regenerate from i to the block end to propose a candidate x′, and accept it with probability

a = min(1, (P_power(x') / P_power(x)) × (q(x|x') / q(x'|x)))

3. Freeze the finalized block, then repeat for the next block with the finalized prefix as context.

Properties: progressive block-by-block construction, global retrospective optimization (any earlier token in the current sequence can be revised), and a compute-for-quality trade-off (much slower than single-pass generation, but tunable).

Experimental results

Three-way comparison: base model vs. GRPO (state-of-the-art RL fine-tuning) vs. Power Sampling:

| Benchmark | Base | GRPO (RL) | Power Sampling | |---|---|---|---| | MATH500 | 45.2% | 58.7% | 58.9% | | HumanEval | 38.4% | 48.6% | 52.3% | | GPQA | 33.7% | 39.5% | 41.2% | | AlpacaEval 2.0 | 17.8% | 28.3% | 31.5% |

Ablations

• α sensitivity: α = 1 reduces to normal sampling (worst); performance rises up to α ≈ 2–4, then can degrade if α is too large — a "sweet spot" exists.
• K sensitivity: K = 0 is worst; accuracy climbs steadily with K until saturation, showing performance is controllable via inference-time compute.
• Diversity (pass@k): GRPO's pass@k curve flattens after k = 1 (diversity collapse), while Power Sampling's keeps rising, indicating it can generate diverse high-quality candidates.

Conclusion

Power Sampling suggests RL fine-tuning largely sharpens rather than adds capability: with an MCMC-based sampler targeting a power distribution, a base model can match or exceed RL-tuned performance on math, coding, and knowledge benchmarks — at the cost of substantially more inference-time computation.