Sleeping Giant: Power Sampling Reveals Your Base Model Is Smarter Than You Think

Forum topic · ✨步子哥 · 2025-12-30

Summary

A Harvard study by Aayush Karan and Yilun Du (arXiv:2510.14901) challenges the belief that reinforcement learning (RL) post-training creates reasoning ability in large language models. The authors argue that RL merely sharpens an existing output distribution—the base model already possesses latent reasoning skills suppressed by divergent sampling. They show mathematically that low-temperature sampling is myopic, weighting tokens by many mediocre futures, whereas a true power distribution (p^alpha) applied to entire sequences favors pivotal tokens leading to correct answers. Since sampling directly from p^alpha is intractable, they introduce an autoregressive Metropolis-Hastings MCMC algorithm with blockwise generation (e.g., 192-token blocks) that iteratively resamples and accepts/rejects continuations. Experiments show dramatic gains: on MATH500, Qwen2.5-Math-7B rises from 49.6% to 74.8%, nearly matching GRPO's 78.5%; on HumanEval, Phi-3.5-mini jumps from 21.3% to 73.2% while GRPO drops to 13.4%; and Pass@k diversity is preserved rather than collapsed. The work signals a shift from training-time to inference-time compute, offering an adjustable intelligence dial without expensive reward models or catastrophic forgetting.

Sleeping Giant: Awakening the Logic Ghost Deep Inside Base Models

> Abstract: In the AI arms race, reinforcement learning (RL) has seemed like a magic wand. But a new Harvard study drops a bombshell: your base model is far smarter than you think. It needs no arduous training and no costly reward model—only the right way to be "awakened." This article dives into an exploration of probability, sampling, and the nature of reasoning, watching mathematics act like a scalpel to open up a large model's neural network and release wisdom suppressed by "greed."

---

Chapter 1: The Misunderstood Genius and the Alchemy Myth

Imagine a scholar who has read every book in human history, versed in everything from quantum mechanics to French cuisine. But when you ask a hard math problem, he starts rambling or gives a plausible-but-wrong answer.

What does mainstream AI training do? It brings in a harsh drill sergeant (RL) with a whip (reward signal) to drill the scholar over and over. Rewards for correct answers, punishment for wrong ones. After endless post-training, the scholar becomes a problem-cracking machine like DeepSeek-R1 or OpenAI o1.

Everyone cheers: RL created the miracle! RL gave the model reasoning!

But Harvard researchers Aayush Karan and Yilun Du raise a heretical question: what if the scholar could already solve the problem?

> Core concept — Distribution Sharpening: The paper's central hypothesis is that RL does not teach the model knowledge it "didn't know"; it merely reshapes the model's output probability distribution—like piling sand spread flat on a table into a sharp tower, so the model only dares to say the "safest" things.

If this holds, RL is not "alchemy" (creating new substance) but "gold panning" (filtering what already exists). The base model, a sleeping giant, has already absorbed logical chains from trillions of training tokens—it is just too "divergent," wandering among countless possibilities and derailed by low-quality thoughts.

What we need is not retraining, but a different way of listening to it.

---

Chapter 2: The Price of Greed — Why We Should Abandon "Temperature"

The Myopia Trap of Low-Temperature Sampling

For a long time, when a model rambled, our only knob was temperature. Lower temperature makes the model conservative, always picking the highest-probability token—supposedly improving accuracy.

Dead wrong. The paper offers a disruptive mathematical proof: low-temperature sampling is not equivalent to a power distribution, and it is thoroughly myopic.

Traditional low-temperature sampling amplifies next-token probability at each step $$t$$ :

p_{\text{temp}}(x_t | x_{<t}) \propto p(x_t | x_{<t})^\alpha

It looks elegant but makes a fatal error: it only looks at its feet, not the future.

> Analogy: Climbing Everest. Low-temperature sampling is a greedy climber who at every step picks the steepest-looking stone ahead, never checking whether a cliff or dead end lies behind it. He ends up stuck on a small mound halfway up—a local optimum.

Power Distribution: Planning with God's-Eye View

The paper argues we want a distribution with global vision: the power distribution, applied to the *entire sequence*:

p_{\text{pow}}(x) \propto p(x)^\alpha

The mathematical difference is subtle but profound (Proposition 1):

Low-temperature sampling weights by: the exponential of the sum of all future path probabilities — $(\sum p_{\text{future}})^\alpha$
Power distribution weights by: the sum of the exponentials of all future path probabilities — $\sum (p_{\text{future}}^\alpha)$

This is like "sum-then-square" vs. "square-then-sum."

Observation 1 reveals the truth:

Low-temperature sampling favors tokens followed by many mediocre paths—even if each path is bad, the total probability is large.
The power distribution favors tokens followed by one path that leads to a brilliant summit.

> Key concept — Pivotal Tokens: In reasoning problems there are pivotal steps: get them right and everything opens up; get them wrong and further effort goes the wrong way. Low-temperature sampling often fails at these points, not daring to pick a lower-probability token that leads to the correct answer. The power distribution, like a Go master, dares to play a seemingly ordinary move for the sake of endgame victory.

---

Chapter 3: The Time-Traveler's Editor — The MCMC Sampling Algorithm

Why hasn't $p^\alpha$ been used before?

Because it's too hard to compute. Sampling from $p(x)^\alpha$ exactly would require enumerating all possible sentences—intractable. We need a scalpel to approximate this perfect distribution: MCMC (Markov Chain Monte Carlo), specifically Metropolis-Hastings.

The Art of Rejecting and Accepting

The authors design an autoregressive MCMC sampling algorithm, like a writer with time-rewind ability:

1. Proposal: The model quickly drafts a passage (e.g., via ordinary low-temperature sampling). 2. Resampling: The algorithm picks a random position $$t$$ mid-passage and cuts off the rest. 3. Regeneration: From position $$t$$ , the model generates a new ending. 4. Accept/Reject: Compare the overall probability score (under $p^\alpha$ ) of the new vs. old ending. If the new one is better—or slightly worse but lucky (random dice roll)—accept it; otherwise reject and roll back.

This iterates at inference time. Every accept/reject is a correction of the chain of logic.

> Code intuition: Without MCMC, if the model writes one wrong line of code (e.g., reversed variable names), everything after is confabulated on the error, crashing the program (hallucination). With MCMC, when the algorithm detects that a choice lowers the coherence (likelihood) of the whole program, it backs up and tries another path until the logic fits seamlessly.

Blockwise Generation

For efficiency, MCMC is not run token-by-token but in blocks:

Set a block size $$B$$ (e.g., 192 tokens).
Generate the first $$B$$ tokens, repeatedly running MCMC refinement within that range until satisfactory.
Fix that portion and generate the next block.

This avoids exponential compute blowup and lets the model consolidate each reasoning step before moving on.

---

Chapter 4: The Data Doesn't Lie — More Than a Tie

Experiments span three domains: math (MATH500), code (HumanEval), and science QA (GPQA).

1. It can crack problems too — without tunnel vision

On MATH500, RL's home turf (e.g., GRPO):

Qwen2.5-Math-7B (Base): 49.6%
GRPO (RL post-trained): 78.5%
Power Sampling (Ours): 74.8%

Merely changing the sampling strategy lifts the base model from failing to excellent, nearly matching a purpose-trained RL model—proof that the reasoning ability really is inside the base model, just buried.

2. Out-of-domain: generalization crushes it

RL models often overfit: trained long on math, they stumble on code or general QA. On HumanEval:

Phi-3.5-mini (Base): 21.3%
GRPO (RL): 13.4% (yes—RL made it *dumber*! Catastrophic forgetting / distribution collapse.)
Power Sampling (Ours): 73.2% 🤯

A +51.9% jump! RL partly "robs Peter to pay Paul," trading general creativity for problem-cracking. Power Sampling is a lossless enhancement, preserving the base model's breadth and flexibility while making its output more rigorous.

3. The victory of diversity

In Figure 5's Pass@k curves:

GRPO: the curve flattens fast—100 samples may yield 99 identical answers. Imagination lost (mode collapse).
Power Sampling: the curve keeps rising—more sampling uncovers more distinct, correct solution paths.

For code generation or creative writing, we don't want an answer-memorizing robot; we want a thinker that approaches problems from multiple angles.

---

Chapter 5: Deep Case Studies — How Does the AI Think?

Case 1: The Modular Arithmetic Trap (MATH500)

Problem: *Solve $n \equiv -11213141 \pmod{18}$ with $0 \le n < 18$ .*

GRPO (RL): It takes a shortcut—the "digit sum" trick (a number's remainder mod 9 equals its digit sum mod 9)—and wrongly applies it to division by 18. Result: wrong answer 4.
Power Sampling (Ours): It dutifully performs long division. It considered the tempting digit-sum trick, but resimulation (MCMC resampling) showed the verification logic wouldn't line up (likelihood drops), so it rejected the impulse. Result: correct answer 13.

Case 2: The Fibonacci Variant (HumanEval)

Problem: *Write a function computing the Fib4 sequence without recursion.*

GRPO (RL): It collapses entirely—wrong loop logic, even variable-update order botched, likely overfit to the standard Fibonacci sequence and confused by the variant.
Power Sampling (Ours): It correctly identifies the "no recursion" constraint, initializes four variables (it's Fib4), and writes correct iterative logic.

These cases illustrate: not just sharper, but steadier. Power Sampling lets the model, when tempted by plausible shortcuts, project forward, realize "this path doesn't work," and pull back from the cliff.

---

Chapter 6: Conclusion — A Transfer of Compute

The Dawn of Inference-Time Compute

This paper isn't just an algorithm; it marks a paradigm shift from training-time compute to inference-time compute. We used to think models got stronger by feeding more data and burning more GPUs (Scaling Laws v1.0). Now DeepSeek-R1, OpenAI o1, and this paper tell us: letting the model "think a bit longer" before speaking may work better.

Power Sampling's essence is trading compute for quality:

Need better answers? Run more MCMC iterations (increase $N_{MCMC}$ ).
Need speed? Reduce iterations.

RL-post-trained models are fixed—you can't dynamically adjust their intelligence at runtime. Power Sampling gives you an adjustable "intelligence dial."

Final Thoughts

"Your base model is smarter than you think" is both praise for AI and a reminder to researchers: before rushing to operate on AI (RL/SFT), perhaps we should first learn to use it correctly.

The sparks of logic already lie within those seemingly chaotic probability distributions. Our job is not to create the fire, but to use mathematics' bellows to gently blow away the ash and let the flame of wisdom rise on its own.

> Actionable insights for developers: > 1. Don't rush to fine-tune: without a high-quality chain-of-thought dataset, blind RL or SFT may be worse than optimizing your sampling strategy. > 2. Watch MCMC: currently slow, but as inference hardware improves, such algorithms may become standard for high-precision reasoning. > 3. Diversity matters: for creative or exploratory code generation, avoid over-RL'd models; a base model plus advanced sampling may be the better choice.

---

*References*: 1. *Karan, A., & Du, Y. (2025). Reasoning with Sampling: Your Base Model is Smarter Than You Think. arXiv preprint arXiv:2510.14901.* 2. *Shao, Z., et al. (2024). DeepSeek-Math: Advancing Mathematical Reasoning through Step-by-Step Exploration.* 3. *Guo, D., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.*