GFT: From Imitation to Reward Fine-Tuning — Why SFT Is Misused, Not Broken

Forum topic · 小凯 · 2026-06-15

Summary

GFT (Group Fine-Tuning) is a new post-training method from Zhejiang University's OmniAI Group (ACES Lab) that reframes supervised fine-tuning (SFT) as a degenerate form of reinforcement learning. The paper, arXiv:2604.14258, identifies two flaws in standard SFT: single-path imitation that causes entropy collapse, and inverse-probability weights that trigger gradient explosions on rare tokens. GFT fixes both with Group Advantage Learning (GAL), which builds per-query groups of 8 responses mixing expert demonstrations, teacher distillations, and self-generated samples with standardized advantages, and Dynamic Coefficient Rectification (DCR), which caps unstable gradient weights for low-confidence tokens. On Qwen2.5-Math-1.5B, GFT surpasses 100k-sample SFT and even GRPO using only 10k queries (80k samples), with ablations showing both components contribute synergistically. GFT also reduces catastrophic forgetting on general benchmarks like MMLU-STEM and improves the SFT-to-GRPO pipeline, resolving the known synergy dilemma. Code is available at https://github.com/ZJU-OmniAI/GFT.

GFT: SFT Is Not a Sin — It's Just Being Used Wrong

> Paper: GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification > arXiv: 2604.14258 | April 2026 > Institution: Zhejiang University OmniAI Group (ACES Lab) > Code: https://github.com/ZJU-OmniAI/GFT

TL;DR

GFT shows that SFT's fundamental problem is not imitation learning itself, but the crude way imitation is done. By upgrading single-expert trajectories to group contrastive learning (GAL) plus dynamic gradient stabilization (DCR), GFT crushes standard SFT with 1/10 the data and provides a better cold start for downstream RL, breaking the "SFT→RL synergy dilemma."

The Two Sins of SFT

The paper re-diagnoses SFT from an overlooked angle: SFT is actually a degenerate form of reinforcement learning. Writing the SFT gradient in policy-gradient form:

\nabla_\theta \mathcal{L}_{\text{SFT}} = -\mathbb{E}\left[ \frac{\mathbb{I}[y=y^*]}{\pi_\theta(y|x)} \nabla_\theta \log \pi_\theta(y|x) \right]

Sin 1: Single-path dependence → entropy collapse. The indicator $\mathbb{I}[y=y^*]$ gives an extremely sparse reward (1 only for exact copying of the expert). The model can copy but not compare — exploration collapses, shrinking the exploration budget available to downstream RL.

Sin 2: Inverse-probability weights → gradient explosion. The weight $1/\pi_\theta(y|x)$ blows up for low-probability tokens — e.g., when the expert uses tokens unfamiliar to the model — causing huge gradient variance, rote memorization, overfitting, and catastrophic forgetting.

This yields the synergy dilemma: SFT alone improves but covers pretraining knowledge; GRPO alone improves; but SFT→GRPO underperforms GRPO alone.

The Two Surgical Tools of GFT

1. GAL (Group Advantage Learning)

Instead of one expert trajectory, GAL builds a mixed response group per query (K=8):

| Source | Count | Role | |--------|-------|------| | Expert demonstration | 1 | Anchors correctness | | Teacher distillation | 3 | Diverse reasoning paradigms | | Model self-generated | 4 | On-policy feedback, corrects internal errors |

Standardized advantage:

A(y_k) = \frac{R(y_k) - \mu(\mathcal{G}_x)}{\sigma_R(\mathcal{G}_x) + \epsilon}

The contrast: traditional SFT says "this trajectory is right, copy it" (absolute, sparse, single), while GAL says "this trajectory is this much better/worse than the group average" (relative, dense, comparative). GAL doesn't remove expert data — it refuses to rely on expert data *only*.

2. DCR (Dynamic Coefficient Rectification)

DCR adaptively clips the unbounded weight $1/\pi_t$ when $\pi_t \to 0$ :

C(\pi_t) = \begin{cases} \text{sg}(\pi_t) & \text{if } \pi_t < \tau \quad \text{(low confidence, blocks explosion)} \\ 1 & \text{if } \pi_t \geq \tau \quad \text{(normal learning)} \end{cases}

For low-confidence tokens, the effective coefficient drops from $1/\pi_t \to \infty$ to $C(\pi)/\pi \approx 1$ — gradient explosion is eliminated at the root, with threshold $\tau \approx 0.7$ empirically validated.

3. Unified Objective

\nabla_\theta \mathcal{L} = \mathbb{E}_{y_k \in \mathcal{G}_x} \left[ A(y_k) \frac{C(\pi_\theta(y_k|x))}{\pi_\theta(y_k|x)} \nabla_\theta \log \pi_\theta(y_k|x) \right]

Combining standardized advantages (GAL), stabilized weights (DCR), and the standard policy gradient.

Experiments: 10× Data Efficiency

Main results (Qwen2.5-Math-1.5B):

| Method | AMC23 | College Math | MATH | Minerva | TabMWP | |--------|-------|-------------|------|---------|--------| | Base | 30.16 | 24.30 | 46.54 | 10.51 | 24.55 | | +SFT (100k samples) | 31.25 | 36.45 | 60.66 | 23.99 | 79.34 | | +GRPO | 44.84 | 35.58 | 65.97 | 21.17 | 76.94 | | +DFT | 36.40 | 38.76 | 64.35 | 23.75 | 82.08 | | +GFT (10k queries = 80k samples) | 46.09 | 40.51 | 70.50 | 28.93 | 85.24 |

Key findings:

GFT outperforms 100k-sample SFT with 1/10 the data
GFT even beats GRPO (46.09 vs 44.84 on AMC23)
The mix isn't the cause: GFT(no mix) ≈ GFT and SFT(mix) ≈ SFT — the gain comes from the mechanism

Ablations confirm synergy: removing both components reduces GFT to SFT (31.25 AMC23); GAL-only reaches 42.81, DCR-only 35.78, full GFT 46.09.

Compatibility with RL

SFT → GRPO: mediocre, poor synergy
GFT → GRPO: better — GAL preserves exploration
SFT → GFT → GRPO: best ceiling. SFT gives a reliable init and format alignment; GFT restores exploration and prevents distribution drift; GRPO exploits high-quality trajectories.

Catastrophic Forgetting (LLaMA-3.2-3B)

| Method | MAWPS | SVAMP | MMLU-STEM | |--------|-------|-------|-----------| | Base | 96.06 | 86.36 | 41.03 | | +SFT | 91.97 (-4.09) | 78.73 (-7.63) | 35.05 (-5.98) | | +GRPO | 94.60 (-1.46) | 88.11 (+1.75) | 39.48 (-1.55) | | +GFT | 95.79 (-0.27) | 84.65 (-1.71) | 43.89 (+2.86) |

GFT barely forgets and even improves MMLU-STEM; KL-divergence analysis confirms GFT's KL is near GRPO's and far below SFT's.

Why GFT Works

1. Repositioning SFT: GFT's key theoretical contribution is redefining SFT from a standalone stage into a degenerate form of RL — the fix is to repair the optimization, not discard imitation. 2. The contrast effect: like a student seeing multiple solutions (right and wrong, simple and complex) rather than one answer key, group learning teaches *what is correct and why*. Teacher-distilled samples break the illusion that there's only one correct way to write an answer. 3. DCR's sense of boundaries: high-confidence tokens learn normally; low-confidence tokens get stabilized without losing the learning signal. 4. Relation to variants: DFT uses diverse responses but no group contrast or gradient rectification; ASFT uses prototypes without standardized advantages; GRPO is pure RL without expert anchoring. GFT unifies imitation and reinforcement.

Limitations

1. Group construction cost: K=8 responses per query makes inference cost 8× SFT per pass, even with 1/10 the training data. 2. Teacher dependency: biased or erroneous teacher samples propagate into the group. 3. Task coverage: validation is mainly on math reasoning; creative writing, dialogue, and code generation remain unverified. 4. Threshold generality: $\tau=0.7$ is optimal for math; adaptive thresholds could help.