A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design (Q-target Framework)

Forum topic · 小凯 · 2026-06-11

Summary

This arXiv paper (2606.11189) by Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, and Cho-Jui Hsieh reinterprets supervised fine-tuning (SFT) as target distribution design. Standard SFT maximizes the likelihood of every token in a demonstrated trajectory, but observed tokens can be non-unique, noisy, or misaligned with the pretrained model's knowledge prior, making strict one-hot fitting suboptimal. The authors introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternative tokens. This perspective unifies many existing SFT variants as implicit selections of a token-level target distribution Q. Building on the framework, the proposed Target-SFT method consistently outperforms baselines across 10 reasoning dataset-model settings, offering a principled lens for designing SFT objectives.

Paper Overview

Field: NLP Authors: Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, Cho-Jui Hsieh arXiv: 2606.11189

Abstract

Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match. We introduce the Q-target framework, which decomposes SFT supervision into two explicit choices:

1. How strongly to rely on the observed token 2. How to allocate the remaining probability mass over alternatives

This perspective unifies many existing SFT variants as implicit choices of target distribution Q. The proposed Target-SFT consistently outperforms baselines across 10 reasoning dataset-model settings.

Key Takeaways

SFT is reframed from loss-objective engineering to token-level target distribution design
The Q-target framework provides a unified view of existing SFT variants
Target-SFT shows consistent empirical gains on 10 reasoning benchmarks

--- *Auto-collected on 2026-06-11*