Select to Think: Letting Small Language Models Pick Better Answers at Critical Moments

This post discusses the paper "Select to Think: Unlocking SLM Potential with Local Sufficiency" by Wenxuan Ye, Yangyang Zhang, and Xueli An (arXiv:2604.26940, released 2026-04-29), in the field of NLP / small language models / reasoning enhancement.

The Problem

Small language models (SLMs, ~1.5B parameters) lag far behind large language models (LLMs) on reasoning tasks. Two existing remedies both fall short:

• Calling an LLM at divergence points: When the SLM gets stuck, invoke an LLM to generate the next tokens. This adds latency, API cost, and privacy risk—and the SLM often can't tell when it's wrong, so it wastes computation before asking for help.
• Knowledge distillation: Train the SLM on LLM-generated outputs. But a 1.5B model cannot faithfully imitate a 32B model's generation distribution—it memorizes the "shape" without the "principle," and fails on genuinely novel reasoning problems.

The Key Finding

The authors inspected what happens at the SLM's reasoning divergence points and found that the token preferred by the LLM is already in the SLM's top-8 candidates 95% of the time. In other words, the SLM's knowledge is stronger than its selection ability—it usually "thinks of" the right answer but ranks it wrong. This reframes the problem: instead of adding knowledge (blocked by capacity limits), improve the SLM's *choice* among candidates it already generates. Selection is a much easier learning task than generation.

The Method: S2T and S2T-LOCAL

S2T changes the LLM's role from ghostwriter to editor: the SLM generates top-K candidates at each position, and the LLM selects the best one rather than writing a new answer. Benefits:

1. The supervision signal becomes a simple discrete ranking/classification problem instead of imitating a complex continuous generation distribution. 2. LLM calls become sparse and precise—only at divergence points—cutting cost and latency, and preserving output consistency with the SLM's own style.

S2T-LOCAL internalizes this ability. During training, pairs of (SLM candidates, LLM rankings) are collected via S2T; the SLM then learns a re-ranking module to score its own candidates. At inference, no LLM is needed at all—analogous to a writer who apprentices under an editor and then applies the editor's judgment independently.

Results

• Under greedy decoding, S2T-LOCAL improves over baseline SLMs by 24.1% on average.
• It matches 8-path self-consistency while using only single-path compute.
• Top-8 candidates from a 1.5B SLM capture the 32B LLM's preferred token 95% of the time.

A 24.1% gain is a qualitative jump, moving small models from "unusable" toward "usable" for reasoning tasks.

Critical Analysis (Feynman-style)

• Understanding vs. selection: Re-ranking learns the LLM's statistical preferences, not causal understanding. The paper does not test out-of-distribution generalization, so the gains may degrade on shifted distributions.
• Missing breakdowns: The 95% capture rate is an average; the paper does not report how it varies across domains and difficulty levels. If capture drops to, say, 70% on genuinely hard problems, S2T may help least exactly where help is most needed.
• Naming caution: "Local Sufficiency" is an empirical regularity, not a theoretically proven sufficiency property—naming is not understanding.
• In fairness: The paper includes extensive comparisons and ablation studies, which is responsible engineering validation of a simple core idea.

Bigger Picture

S2T suggests capability can be amplified through *selection* rather than only through *scale*, challenging the assumption that reasoning quality requires ever-larger models. Its generate-then-re-evaluate structure echoes human dual-process cognition (fast intuition followed by slow deliberation), pointing toward future small models with lightweight metacognition—knowing when to trust and when to double-check their first instinct.

References

1. Ye, W., Zhang, Y., & An, X. (2026). *Select to Think: Unlocking SLM Potential with Local Sufficiency*. arXiv:2604.26940. 2. Hinton, G., et al. (2015). *Distilling the Knowledge in a Neural Network*. arXiv:1503.02531. 3. Wei, J., et al. (2022). *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*. NeurIPS 2022. 4. Kahneman, D. (2011). *Thinking, Fast and Slow*. Farrar, Straus and Giroux. 5. Feynman, R. P. (1974). *Cargo Cult Science*. Caltech Commencement Address.