Select to Think: How LLM-Guided Re-Ranking Unlocks Small Language Model Reasoning

> Paper: Select to Think: Unlocking SLM Potential with Local Sufficiency > Authors: Wenxuan Ye, Yangyang Zhang, Xueli An > arXiv: 2604.26940 > Published: 2026-04-29 > Field: NLP / Small Language Models / Reasoning Enhancement

Key points

• The core finding: At SLM reasoning failure points (divergence points), the token preferred by an LLM is already in the SLM's top-8 candidate list 95% of the time. The SLM's *knowledge* is stronger than its *selection ability* — it "thinks of" the right answer but ranks it wrong.
• Why existing fixes fail:
• *LLM fallback / token handoff*: calling an LLM at divergence points incurs latency, API cost, and privacy risks, and SLMs cannot reliably detect when they need help.
• *Knowledge distillation*: a 1.5B model cannot faithfully approximate a 32B model's generation distribution; it memorizes "shape" without "principle."

The S2T method

Don't let the LLM write answers for the SLM — let it select among the SLM's answers.

1. The SLM generates its top-K candidates at each position (e.g., top-8). 2. The LLM's job is *re-ranking* those candidates, not rewriting — like an editor marking up a draft rather than a ghostwriter replacing it. 3. Benefits:

• The supervision signal becomes a discrete, low-dimensional ranking task instead of matching a full generation distribution — much easier for a small model to learn.
• LLM invocations become sparse and precise, limited to divergence points, cutting cost and latency.
• Output continuity and style are preserved since the LLM only picks from the SLM's own output space.

S2T-LOCAL: self-re-ranking without the LLM

During training, S2T-style data (SLM candidates + LLM rankings) is used to teach the SLM a re-ranking module. At inference, the SLM generates candidates and re-ranks them itself — no LLM needed. This mirrors a "generate, then evaluate" dual-process structure analogous to Kahneman's System 1 / System 2.

Reported results:

• Average 24.1% improvement over baseline SLMs under greedy decoding
• Matches 8-path self-consistency performance with only single-path compute
• 1.5B SLM top-8 candidates capture the 32B LLM's choice 95% of the time

Critical review (Feynman-style)

• Understanding vs. correct selection: S2T-LOCAL learns the LLM's *preference patterns* — statistical imitation, not causal understanding. Out-of-distribution generalization is not explicitly tested.
• Cargo-cult risk: the 95% top-8 capture rate is an average; the paper lacks a breakdown by domain and difficulty. Hard reasoning problems may have far lower capture rates.
• "Local Sufficiency" naming: this is an empirical observation ("the right answer is usually in the top-8"), not a theoretically proven sufficiency property; the name may overstate its status.
• Presentation: the core intuition is simple and demonstrable; the paper compensates with extensive ablations and comparisons — thorough, though the volume of engineering detail may mask the idea's simplicity.

Bigger picture

S2T suggests capability can be amplified through selection, not just scale — a lightweight form of metacognition where the model learns to "check its first reaction." It parallels human intuition-then-reflection reasoning, and hints at future research into richer reflection: generating counterexamples, questioning assumptions, and knowing when to seek help.

References

1. Ye, W., Zhang, Y., & An, X. (2026). *Select to Think: Unlocking SLM Potential with Local Sufficiency*. arXiv:2604.26940. 2. Hinton, G., et al. (2015). *Distilling the Knowledge in a Neural Network*. arXiv:1503.02531. 3. Wang, X., et al. (2023). *Large Language Models are not Fair Evaluators*. ACL 2023. 4. Wei, J., et al. (2022). *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*. NeurIPS 2022. 5. Kahneman, D. (2011). *Thinking, Fast and Slow*. Farrar, Straus and Giroux. 6. Feynman, R. P. (1974). *Cargo Cult Science*. Caltech Commencement Address.