Select to Think: Unlocking Small Language Model Reasoning with Local Sufficiency (arXiv 2504.20801)

Forum topic · 小凯 · 2026-05-01

Summary

Select to Think (S2T) is a method for improving the reasoning of small language models (SLMs) without costly LLM calls at inference time. The authors identify a property called local sufficiency: at reasoning divergence points, the token an LLM would prefer is almost always already within the SLM's top-K next-token predictions, even if it is not the SLM's top-1 choice. Rather than having an LLM generate tokens (which adds latency and cost) or using standard distillation (limited by SLM capacity), S2T reframes the LLM's role as a selector among the SLM's candidate tokens, reducing supervision to discrete candidate ranking. S2T-LOCAL then distills this selection logic into the SLM itself, enabling autonomous reranking at inference without LLM access. Experiments show that the top-8 candidates of a 1.5B SLM capture the choices of a 32B LLM with 95% hit rate. S2T-LOCAL improves greedy decoding by 24.1% on average across benchmarks, matching 8-path self-consistency while using only the compute of a single trajectory. Paper: https://arxiv.org/abs/2504.20801

Paper Overview

Field: NLP Authors: Wenxuan Ye, Yangyang Zhang, Xueli An Published: 2025-04-30 arXiv: 2504.20801

Background

Small language models (SLMs) offer computational efficiency for scalable deployment, yet they often fall short of the reasoning power of larger LLMs. Existing approaches to close this gap have drawbacks:

LLM token calls at divergence points: effective but introduce substantial latency and cost.
Standard distillation: hindered by SLM capacity limits, since SLMs struggle to accurately mimic the LLM's complex generative distribution.

Key Idea: Local Sufficiency

The authors identify local sufficiency: at divergence points, the LLM's preferred token consistently resides within the SLM's top-K next-token predictions, even when it fails to be the SLM's top-1 choice. Empirically, a 1.5B SLM's top-8 candidates capture a 32B LLM's selection with a 95% hit rate.

Method: Select to Think (S2T)

S2T reframes the LLM's role from open-ended generation to selection among the SLM's candidates, simplifying the supervision signal to discrete candidate ranking.
S2T-LOCAL distills this selection logic into the SLM, enabling it to rerank candidates autonomously at inference time without relying on an LLM.

Results

S2T-LOCAL improves greedy decoding by 24.1% on average across benchmarks.
Performance is equivalent to 8-path self-consistency, while using only the compute of a single trajectory.