If you had a rebellious student at home, would you force them to memorize the entire encyclopedia, or teach them how to pick the right answer among four multiple-choice options?
In the AI world, we've been doing the former: using "distillation" to make 1.5B-parameter small language models (SLMs) imitate the deep reasoning distributions of much larger models. The result? The small models struggle, and performance still falls short.
A new paper, "Select to Think" (arXiv:2604.26940), argues the direction is wrong. The key to improving small-model reasoning isn't teaching them to "talk like the big models" — it's teaching them to select the correct answer from their own candidate set.
1. A Surprising Finding: It "Knows" — It Just Doesn't "Choose"
The authors propose an observation called "Local Sufficiency."
Simply put: when a 1.5B model makes an error on a math or logic problem (a wrong Top-1 prediction), researchers found that the answer considered correct by a large model appears among the small model's top-8 candidate tokens about 95% of the time.
Feynman-style analogy: it's like an intern whose final proposal is wrong, but whose brainstorm list already contains the perfect idea — they just lack the judgment to rank it first.
2. From "Generation" to "Selection"
Since the correct answer is already in the candidate pool, why force the small model to fit complex probability distributions?
The S2T (Select to Think) framework downgrades the task:
- Traditional approach: force the small model to replicate every token of the large model's output.
- S2T approach: the large model acts as a "mentor," scoring the correct candidate among the 8 candidates the small model generates itself.
- With no external calls, the 1.5B model's performance on logic-reasoning tasks improved by 24.1%.
- Its reasoning quality matched self-consistency methods that require 8 full passes — while keeping fast, single-pass inference.
This turns a hard "creative task" into an easy "multiple-choice task." For a parameter-limited model, learning to *pick* is far easier than learning to *originate*.
3. S2T-Local: Baking the Mentor Into the Model
The most hardcore contribution: rather than consulting an external guide every time, the authors distill this selection logic into the small model itself via an "Inner Critic."
When the model detects it may be about to go off the rails (triggered via KL divergence detection), it automatically activates "thinking mode": stop, glance at its own candidate tokens, and re-rank them.
The results?
Editorial Take: Intelligence Is Judgment, Not Just Knowledge
The insight here: AI intelligence lies not only in knowledge volume, but in judgment.
We've long assumed small models underperform because their "brain capacity" can't hold enough knowledge. But *Select to Think* shows the knowledge is often already hiding in the crevices of the weights — what's missing is that flash of insight: picking truth from among many distractors at the critical moment.
This "re-ranking" idea may be the first step toward giving on-device AI genuine reflection capability.
Paper reference: arXiv:2604.26940
*What do you think is the most important "judgment skill" for small models to learn? Join the discussion in the comments.*