[论文] Strategically Diverse Sampling for Self-Training

论文概要

研究领域: NLP 作者: Alexander Gurung, Esmeralda S. Whitammer, Mirella Lapata 发布时间: 2026-09-25 arXiv: 2609.31571

中文摘要

包括 RL 和测试时扩展在内的许多 LLM 训练与推理方法都依赖重复采样，但只有响应之间有意义的差异才能带来收益。自训练面临同样的挑战：训练数据通常通过独立同分布（IID）采样并按正确性过滤来构造，这会过度代表模型本就偏好的策略。我们研究“策略多样性”——即解决问题的方法之间存在实质性差异——作为构造自训练数据的替代原则。我们用两种采样方法生成策略多样的数据：GROOT——一种构造方法层次树并采样不同路径的新方法；以及言语化采样（Verbalized Sampling, VS），经适配后产生非结构化的方法集合。在竞技编程和“下一章预测”任务上，用策略采样数据训练的模型在困难任务上优于 IID 训练的对应模型，并为 RL 和测试时扩展提供了强初始化。最引人注目的是，在 Qwen3-4B 的策略多样但错误的轨迹上自训练，居然超过了从 235B 教师模型进行 IID 蒸馏的效果。这些结果挑战了关于“什么构成有用的自训练数据”的流行假设，表明方法的多样性可能比正确性或教师模型的规模更重要。

原文摘要

Many LLM training and inference methods, including RL and test-time scaling, depend on repeated sampling, but benefit only when the responses meaningfully differ. Self-training faces the same challenge: training data is typically constructed by sampling IID responses and filtering primarily for correctness, thereby overrepresenting strategies a model already favours. We investigate strategic diversity, or substantive variation among approaches to a problem, as an alternative principle for constructing self-training data. We generate strategically diverse data with two sampling methods: GROOT, a new method which constructs a hierarchical tree of approaches and samples distinct paths, and Verbalized Sampling (VS), adapted to produce an unstructured set of approaches. Across competitive progr...

#论文 #arXiv #NLP #小凯