Select to Think (S2T): Teaching 1.5B Small Models to Pick the Right Answer Instead of Memorizing

Forum topic · QianXun · 2026-05-01

Summary

A forum post discusses the paper "Select to Think" (arXiv:2604.26940), which challenges conventional knowledge distillation for small language models (SLMs). The authors observe a property called Local Sufficiency: when a 1.5B-parameter model's top-1 prediction is wrong, the correct answer still appears within its top-8 candidates about 95% of the time. Rather than forcing small models to mimic a large model's full output distribution, the proposed S2T (Select to Think) framework reframes distillation as a selection task: a large model acts as a tutor that scores the correct candidate among the small model's own top-8 proposals. The variant S2T-Local distills this selection ability into an "Inner Critic" inside the small model, triggered by KL-divergence detection, letting the model pause and re-rank its candidates. Reported results: a 24.1% improvement on logical reasoning for a 1.5B model with no external calls, matching self-consistency (8-sample) quality while keeping single-pass inference speed. The post argues that small models often already contain the needed knowledge and mainly lack judgment—selecting the right answer among candidates—rather than capacity.

Introduction

If you had a rebellious student at home, would you force them to memorize an entire encyclopedia, or teach them to accurately pick the correct answer among four multiple-choice options?

In AI, we have been doing the former: through distillation, we try to make 1.5B-parameter small language models (SLMs) imitate the deep, inscrutable reasoning distributions of much larger models. The result? The small models struggle, and performance still falls short.

The paper "Select to Think" (arXiv:2604.26940) argues the direction is wrong: improving small-model reasoning is not about teaching it to "talk like the big model," but about teaching it to *select* the right option among its own candidate predictions.

---

1. A Surprising Finding: It Already "Knows"—It Just Doesn't "Choose"

The authors propose an intriguing observation called "Local Sufficiency."

In plain terms: when a 1.5B model gets a math or logic problem wrong (wrong Top-1 prediction), researchers examining its "scratch work" found that the answer a large model considers correct has about a 95% probability of being within the small model's top-8 candidate tokens.

Feynman-style analogy: it's like an intern whose final proposal is wrong, but whose brainstorm list already contains the perfect idea—they simply lack the judgment to rank it first.

2. From Generation to Selection

If the correct answer is already in the candidate pool, why force the small model to fit complex probability distributions?

The S2T (Select to Think) framework "downgrades" the task:

Traditional approach: force the small model to replicate the large model's every token.
S2T approach: the large model acts as a "tutor," assigning a high score to the correct option among the 8 candidates the small model itself generates.

A complex "generation task" thus becomes a simple "multiple-choice task." For parameter-limited small models, learning to *pick* is much easier than learning to *create*.

3. S2T-Local: Internalizing the Tutor

The most substantial contribution: the authors do not require consulting an external guide at inference time. Instead, S2T-Local distills the large model's selection logic into an "Inner Critic" inside the small model.

When the model detects it may be about to go off-track (triggered via KL-divergence detection), it automatically enters "thinking mode": it pauses, reviews its candidate tokens, and re-ranks them.

Results: without any external calls, a 1.5B model improved by 24.1% on logical reasoning tasks. Its reasoning quality matched "self-consistency" methods that require 8 inference passes—while retaining fast single-pass inference speed.

---

Editorial Take

The insight here: AI intelligence lies not only in knowledge volume, but in judgment.

We long assumed small models underperform because of insufficient "brain capacity." "Select to Think" shows the knowledge is often already embedded in the weights—what's missing is the spark of choosing truth among distractors at the critical moment. This re-ranking idea may be a first step toward giving on-device AI genuine "reflection" capability.

What do you think is the most important "judgment skill" for small models to learn? Share your thoughts in the comments.

---

Paper: arXiv:2604.26940