Power Sampling: Your Base Model Is Smarter Than You Think — How MCMC at Inference Time Rivals RL Post-Training

Forum topic · ✨步子哥 · 2025-12-30

Summary

A Harvard study by Aayush Karan and Yilun Du (arXiv:2510.14901) argues that reinforcement learning (RL) post-training does not teach base models new reasoning abilities, but merely sharpens their output distributions—a hypothesis called distribution sharpening. The paper shows that low-temperature sampling is a poor approximation of the desired power distribution p(x)^α: it is myopic, weighing tokens by the sum of future path probabilities raised to α rather than the sum of α-powered future probabilities, causing failures at pivotal reasoning tokens. The authors propose autoregressive MCMC sampling (Metropolis-Hastings with blockwise regeneration) to approximate p(x)^α at inference time. On MATH500, Qwen2.5-Math-7B with power sampling rises from 49.6% to 74.8%, nearly matching GRPO's 78.5%. On HumanEval, Phi-3.5-mini jumps from 21.3% (base) to 73.2%, while GRPO degrades to 13.4%, showing power sampling preserves out-of-domain generality and diversity (Pass@k) where RL causes mode collapse. The results support a shift from training-time to inference-time compute for reasoning.

Power Sampling: Waking the Logical Ghost Inside Base Models

A detailed Chinese forum post discusses a Harvard study (Karan & Du, 2025, arXiv:2510.14901) arguing that base LLMs already contain latent reasoning ability, and that a sampling algorithm—not RL training—can unlock it.

Key points

1. RL as "distribution sharpening," not alchemy

RL post-training (e.g., GRPO, as in DeepSeek-R1 / OpenAI o1) is widely credited with *creating* reasoning ability.
The paper's core hypothesis, distribution sharpening, claims RL only reshapes the model's probability distribution toward high-probability modes—it filters existing capabilities rather than adding new ones. RL is "gold panning," not alchemy.

2. Low-temperature sampling is the wrong way to sharpen

Low-temperature sampling applies $p_{\text{temp}}(x_t | x_{<t}) \propto p(x_t | x_{<t})^\alpha$ at each step.
The desired power distribution acts on whole sequences: $p_{\text{pow}}(x) \propto p(x)^\alpha$ .
Proposition 1 shows the difference: low temperature weights a token by $(\sum p_{\text{future}})^\alpha$ , while the power distribution weights by $\sum (p_{\text{future}}^\alpha)$ .
Consequence: low temperature favors tokens with many mediocre continuations; the power distribution favors tokens with even one path to a correct answer—crucial at pivotal tokens in reasoning chains.

3. Autoregressive MCMC sampling

Since sampling exactly from $p(x)^\alpha$ is intractable, the authors use Metropolis-Hastings in an autoregressive loop:

1. Generate a draft segment (e.g., with low-temperature sampling). 2. Pick a random position $$t$$ , truncate, and regenerate a new continuation. 3. Accept/reject the new ending based on the $p^\alpha$ likelihood score. 4. Iterate at inference time; generation proceeds in blocks (e.g., B = 192 tokens) for efficiency.

4. Experimental results

| Task | Base model | RL (GRPO) | Power Sampling | |---|---|---|---| | MATH500 (Qwen2.5-Math-7B) | 49.6% | 78.5% | 74.8% | | HumanEval (Phi-3.5-mini) | 21.3% | 13.4% | 73.2% (+51.9%) | | GPQA | tested | — | competitive |

Power sampling nearly matches RL in-domain (math) without any training.
Out-of-domain (code), RL actually *degrades* the base model (catastrophic forgetting / distribution collapse), while power sampling massively improves it—a "lossless" enhancement.
Pass@k curves: GRPO's curve flattens (mode collapse), while power sampling keeps finding diverse correct solutions.

5. Case studies

Modular arithmetic (MATH500): GRPO misapplies the digit-sum shortcut to modulus 18 and answers 4; power sampling rejects the shortcut after resampling (future likelihood drops) and does long division, answering 13 correctly.
Fib4 coding task (HumanEval): GRPO writes broken loop logic (overfit to standard Fibonacci); power sampling respects the no-recursion constraint and initializes four variables correctly.

6. Implications

A paradigm shift from training-time compute to inference-time compute: quality is bought with more MCMC iterations, giving a tunable "intelligence knob" that fixed RL-post-trained models lack.
Practical advice from the post:

1. Don't rush to fine-tune—optimize sampling strategy if you lack high-quality CoT data. 2. Watch MCMC methods; they may become standard for high-precision inference as hardware improves. 3. For creative or exploratory tasks, prefer base models with advanced sampling over heavily RL-tuned models.

References

1. Karan, A., & Du, Y. (2025). Reasoning with Sampling: Your Base Model is Smarter Than You Think. arXiv:2510.14901 2. Shao, Z., et al. (2024). DeepSeek-Math. 3. Guo, D., et al. (2025). DeepSeek-R1.