Paper Overview
- Field: NLP
- Authors: Wenxuan Ye, Yangyang Zhang, Xueli An
- Published: 2025-04-30
- arXiv: 2504.20801
- The top-8 candidates of a 1.5B SLM capture the 32B LLM's choice with a 95% hit rate.
- S2T-LOCAL improves greedy decoding by 24.1% on average across benchmarks.
- This matches the performance of 8-path self-consistency while using only single-trajectory compute cost.
Summary
Small language models (SLMs) offer computational efficiency for scalable deployment, yet they often fall short of the reasoning power of large language models (LLMs). Current approaches to close this gap invoke an LLM to generate tokens at points of reasoning divergence, but these external calls introduce substantial latency and cost. Standard distillation, meanwhile, is hindered by capacity limitations, as SLMs struggle to accurately mimic the LLM's complex generative distribution.
The authors resolve this dilemma by identifying local sufficiency: at divergence points, the LLM's preferred token consistently resides within the SLM's top-K next-token predictions, even when it fails to emerge as the SLM's top-1 choice.
Based on this insight, they propose Select to Think (S2T), which reframes the LLM's role from open-ended generation to selecting among the SLM's candidates, reducing the supervision signal to discrete candidate ranking. They further introduce S2T-LOCAL, which distills this selection logic into the SLM so it can autonomously rerank candidates at inference time without relying on an LLM.
Key Results
*Auto-collected on 2026-05-01.*