A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design (Q-target Framework)

Forum topic · 小凯 · 2026-06-11

Summary

Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory, but observed tokens can be non-unique, noisy, or misaligned with the pretrained model's knowledge prior, making strict one-hot fitting suboptimal. This paper, arXiv:2606.11189 by Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, and Cho-Jui Hsieh, reinterprets SFT as target distribution design: rather than studying only the loss objective, it analyzes the token-level target distribution the loss drives the model to match. The authors introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: how strongly to rely on the observed token, and how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit choices of a target distribution Q. Building on this insight, the authors propose Target-SFT, an explicit target distribution method. Experiments across 10 reasoning dataset-model settings show that Target-SFT consistently outperforms baseline SFT approaches, demonstrating the value of deliberate target distribution design over implicit one-hot supervision.

Paper Overview

Field: NLP Authors: Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, Cho-Jui Hsieh Published: 2026-06-09 arXiv: 2606.11189

Abstract

Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior.

In this work, the authors reinterpret SFT as target distribution design: instead of studying only the loss objective, they analyze the token-level target that the loss drives the model to match.

Key Ideas

Q-target framework: decomposes SFT supervision into two explicit choices:
Unification: this perspective unifies many existing SFT variants as implicit choices of a target distribution Q.
Target-SFT: a method with an explicitly designed target distribution that consistently outperforms baselines across 10 reasoning dataset-model settings.

Takeaway

Treating SFT as an exercise in designing the target distribution — rather than blindly fitting one-hot token targets — yields more robust fine-tuning, especially for models with strong pretrained knowledge priors.

--- *Auto-collected on 2026-06-11*