[论文] A Unifying Lens on Supervised Fine-Tuning Through Target Distribution ...

论文概要

研究领域: NLP
作者: Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, Cho-Jui Hsieh
发布时间: 2026-06-09
arXiv: 2606.11189

中文摘要

监督微调（SFT）通常最大化演示轨迹中每个token的似然，但观察到的token可能不唯一、有噪声或与模型先验不对齐。本文将SFT重新诠释为目标分布设计：分析损失驱动模型匹配的token级目标。提出Q-target框架，将SFT监督分解为两个选择：多大程度上依赖观察token，以及如何在替代方案上分配剩余概率质量。统一了多种现有SFT变体为目标分布Q的隐式选择。提出的Target-SFT在10个推理数据集-模型设置上持续优于基线。

原文摘要

Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match. We introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit cho...

自动采集于 2026-06-11

#论文 #arXiv #NLP #小凯