SGUID: Selecting a Compact Skill Bank for Stable Model-Skill Co-Evolution in LLM Distillation

Forum topic · 小凯 · 2026-10-12

Summary

SGUID is a method for selecting a compact subset of skills before distilling them into large language models. Skills are reusable procedural guidance added at inference time, and prior work retrieves them by semantic relevance for distillation or inference-time patching, largely ignoring each skill's individual utility. The authors show that in on-policy distillation with skill-conditioned teachers, fewer than 25% of retrieved skills provide useful distillation signals. SGUID keeps a skill only if it consistently yields effective learning signals during training. Across four models from the Olmo and Qwen families, distilling just 6 selected skills matched or exceeded full-bank distillation on three of four models, and adding 3 newly selected skills in a second round surpassed full-bank distillation on all four—while the full bank is up to 11x larger. SGUID also enables stable model-skill co-evolution: after each distillation round, a new candidate bank is curated from the updated model's rollouts and SGUID selects the next skills. This lifted Qwen3-8B from 64.3% to 66.3%, while naive updates with unfiltered skills hurt Qwen3-4B on HMMT25. Paper: arXiv 2610.12367.

Paper Overview

Field: NLP Authors: Yuhan Liu, Xiyao Ma, Zhongkai Sun, Xu Han, Chengyuan Ma, Benjamin Z. Yao, Chenlei Guo Published: 2026-10-08 arXiv: 2610.12367

Abstract

Skills, reusable procedural guidance added at inference, can substantially improve LLM downstream performance. Prior work retrieves skills from a bank by semantic relevance, then uses them as inference-time patches or for model distillation. The individual utility of each skill, however, is largely neglected.

The authors first show that in on-policy distillation—where skill-conditioned policies serve as teachers—fewer than 25% of retrieved skills provide useful distillation signals. They then propose SGUID, a method for selecting a compact subset of skills for distillation: a skill is retained only if it consistently yields effective learning signals during training. The selected skills are then distilled to produce a better model.

Key Results

Not all skills are worth distilling: Across four models from the Olmo and Qwen families, distilling just 6 selected skills matched or exceeded the mean avg@12 of full-bank distillation on three of the four models. A second round with 3 newly selected skills surpassed full-bank distillation on all four models—while the full bank is up to 11x larger.
Stable model-skill co-evolution: After each distillation round, a new candidate bank is curated from the updated model's rollouts, and SGUID selects the next skills to internalize. In the second round, this loop selected 3 new skills, improving Qwen3-8B from 64.3% to 66.3%.
Selection is critical for stability: On Qwen3-4B, naively updating the model with unfiltered skills hurt performance (HMMT25 dropped 0.3 points), whereas SGUID improved HMMT25 by 0.5 points after round one and 1.1 points after round two.

These results establish skill selection as a key mechanism for stable model-skill co-evolution.