UCLA's Q-Target Framework Rethinks Supervised Fine-Tuning: From Loss Functions to Target Distribution Design

Forum topic · 小凯 · 2026-06-10

Summary

Researchers at UCLA (arXiv:2606.11189) propose the Q-target framework, a unifying perspective on supervised fine-tuning (SFT) of large language models. The framework decomposes the SFT target into Q_t = γ_t · δ(y_t) + (1 − γ_t) · π̃_t, where γ_t captures how much to trust each demonstration token and π̃_t specifies where to place the remaining probability mass. The authors show that many existing SFT variants—standard SFT, DFT, Label Smoothing, ProFiT, EAFT, CFT, ASFT, Proximal SFT, GEM, and knowledge distillation—are specific choices of these two parameters. Building on this, they introduce Target-SFT, which sets γ_t to the model's own confidence p_t and uses a teacher-guided geometric mixture as the residual distribution. Across ten dataset-model combinations covering math reasoning and medical tasks, Target-SFT consistently achieves best or near-best performance, while prior variants are inconsistent. The work reframes SFT method design from loss engineering to target distribution design.

UCLA's Q-Target Framework: Reinventing Supervised Fine-Tuning Through Target Distribution Design

> Authors: Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, Cho-Jui Hsieh > Institution: University of California, Los Angeles (UCLA) > arXiv: 2606.11189 > Project page: https://txie1.github.io/Target-SFT/

The Problem with Traditional SFT

Standard supervised fine-tuning (SFT) assumes every token in the demonstration data deserves to be perfectly copied. Its loss function:

\mathcal{L}_{\text{SFT}} = -\log \pi_\theta(y_t \mid x_{＜t})

forces the model to place all probability mass on a single token — matching a one-hot target distribution. But real demonstration data contains noise, ambiguity, and errors, and a pretrained model already has rich prior knowledge. Forcing it to memorize regardless of its own understanding is inefficient and can damage existing capabilities.

The paper's core reframing: SFT is not about loss function design — it is about target distribution design.

The Q-Target Framework

The authors decompose the ideal training target at each position:

Q_t = \gamma_t \cdot \delta_{y_t} + (1 - \gamma_t) \cdot \tilde{\pi}_t

γ_t ∈ [0, 1]: how much we trust the demonstration token y_t (γ_t = 1 recovers standard SFT)
π̃_t: the surrogate distribution used for the remaining probability mass (model's own distribution, a teacher model, a uniform distribution, etc.)

This expands SFT design from a single dimension ("imitate or not") to two dimensions: how much to trust and what to fall back on.

Unifying a Decade of SFT Research

The paper shows nearly all existing SFT variants are specific choices of (γ_t, π̃_t):

| Method | Category | γ_t | π̃_t | |--------|----------|-----|------| | Standard SFT | One-hot | 1 | — | | DFT | Label trust | p_t (model confidence) | Model's own distribution | | Beyond-log | Label trust | p_t^α | Model's own distribution | | ProFiT | Label trust | 1{p_t > τ} (threshold) | Model's own distribution | | CFT | Label trust | Causal criticality | Model's own distribution | | EAFT | Label trust | Entropy-based weights | Model's own distribution | | Label Smoothing | Residual distribution | 1 − λ | Uniform distribution | | SFT + KL | Residual distribution | 1/(1+λ) | Reference model distribution | | ASFT | Residual distribution | p_t/(p_t + λ) | Base model distribution | | Proximal SFT | Residual distribution | Clipping-dependent | Old model distribution | | GEM | Residual distribution | 1 | Annealed model distribution | | Knowledge Distillation | Residual distribution | 0 | Teacher model distribution | | Distillation (Hybrid) | Residual distribution | 1 − λ | Teacher model distribution |

A decade of seemingly separate methods turns out to be exploration within one design space — some tuning trust (γ_t), others designing surrogate distributions (π̃_t), with no one previously naming the unifying framework.

Target-SFT: Listening to the Model's Own Uncertainty

Built on the Q-target view, Target-SFT makes two elegant choices:

1. Trust set by model confidence:

\gamma_t = p_t = \pi_\theta(y_t \mid x_{＜t})

If the model is already confident about the correct token, the demonstration is likely right and worth learning; if the model finds the token highly improbable, forcing it in may be counterproductive.

2. Teacher-guided residual distribution:

\tilde{\pi}_t^{\text{guided}}(a) \propto \pi_\theta(a)^{1-\eta} \cdot \pi_T(a)^{\eta}

A geometric mixture of the model's own distribution and a stronger teacher model π_T, with η controlling teacher influence. The final target:

Q_t^{\text{TARGET}} = p_t \cdot \delta_{y_t} + (1 - p_t) \cdot \tilde{\pi}_t^{\text{guided}}

The design is adaptive (confidence handles everything, no manual thresholds) and symmetric (it combines demonstrations, the model's own prior, and a teacher, weighted automatically by confidence).

Experimental Results

Evaluated on ten dataset–model combinations spanning mathematical reasoning and medical tasks:

Target-SFT achieved the best or near-best results in all settings
Traditional SFT performed well on some settings but lagged significantly on others
Other SFT variants (DFT, Label Smoothing, etc.) were inconsistent

This consistency suggests Target-SFT is a general design principle rather than a task-specific trick.

Why It Works

Consistency with pretraining: p_t encodes pretrained knowledge, naturally balancing priors against demonstrations instead of erasing them.
Bayesian view: SFT is belief updating with evidence. Traditional SFT treats every token as infinitely strong evidence; Target-SFT scales the update by evidence quality and prior confidence.

What the Framework Opens Up

The (γ_t, π̃_t) space supports choices based on model confidence, causal token importance, data quality estimation, task difficulty (curriculum learning), or human feedback. It also suggests better initialization for downstream RLHF/DPO: a model that retains pretrained knowledge while absorbing demonstration value is easier to guide with RL than one that drifted during SFT.

Conclusion

Traditional SFT is a rigid teacher demanding every student memorize the board. Target-SFT is an experienced teacher who asks a better question — not "what did I teach?" but "how much did the student understand?" The Q-target framework shifts SFT from teacher-centered loss engineering to student-centered target distribution design.

Reference

Xie, T., Ban, Y., Hong, Y., An, S., Chen, Y., & Hsieh, C. J. (2026). A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design. *arXiv preprint arXiv:2606.11189*.