GFT: From Imitation to Reward Fine-Tuning—Group Advantage Learning and Dynamic Coefficient Rectification

Forum topic · 小凯 · 2026-06-15

Summary

GFT (Group Fine-Tuning), a paper from Zhejiang University's OmniAI Group (arXiv:2604.14258), reframes supervised fine-tuning (SFT) as a degenerate form of reinforcement learning with two flaws: single-path dependence causing entropy collapse, and inverse-probability weights causing gradient explosion. GFT addresses these with two techniques: Group Advantage Learning (GAL), which builds a mixed response group per query (expert demonstration, teacher distillation, and self-generated samples, K=8) and learns from standardized relative advantages instead of copying a single expert trajectory; and Dynamic Coefficient Rectification (DCR), which adaptively clips gradient coefficients for low-confidence tokens to eliminate gradient explosion. On Qwen2.5-Math-1.5B, GFT matches or exceeds SFT trained on 100k samples using only 10k queries, outperforms GRPO on AMC23 (46.09 vs 44.84), and shows minimal catastrophic forgetting on LLaMA-3.2-3B general benchmarks. GFT also works best as a bridge in SFT→GFT→GRPO pipelines, resolving the known SFT-RL synergy dilemma. Code is available on GitHub.

Overview

GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification

Paper: arXiv: 2604.14258 (April 2026)
Institution: Zhejiang University, OmniAI Group (ACES Lab)
Code: https://github.com/ZJU-OmniAI/GFT

One-line summary: GFT shows that the fundamental problem with SFT is not imitation learning itself, but that imitation is done too crudely. By upgrading single-expert trajectories to group contrastive learning (GAL) plus dynamic gradient stabilization (DCR), GFT beats standard SFT across the board with 1/10 the data and provides a better cold start for downstream RL, breaking the "SFT→RL synergy dilemma."

The Two "Original Sins" of SFT

The paper's key insight: SFT is a degenerate form of reinforcement learning. Writing the SFT gradient in policy-gradient form:

\nabla_\theta \mathcal{L}_{\text{SFT}} = -\mathbb{E}\left[ \frac{\mathbb{I}[y=y^*]}{\pi_\theta(y|x)} \nabla_\theta \log \pi_\theta(y|x) \right]

Sin 1: Single-path dependence → entropy collapse

The indicator $\mathbb{I}[y=y^*]$ gives an extremely sparse reward: only exact replication of the expert earns 1.
The model can only "copy," not "compare" — exploration ability drops to zero, causing entropy collapse.
Aftereffect: the exploration budget for downstream RL is severely compressed.

Sin 2: Inverse-probability weights → gradient explosion

The weight $1/\pi_\theta(y|x)$ grows sharply for low-probability tokens (e.g., when the expert uses unfamiliar tokens).
Result: extreme gradient variance → mechanical memorization → overfitting → catastrophic forgetting.

The synergy dilemma: SFT alone improves but covers pretrained knowledge; GRPO alone improves; but SFT→GRPO pipelines perform *worse* than GRPO alone.

Core Techniques

1. GAL (Group Advantage Learning)

Instead of a single expert trajectory, GFT builds a mixed response group per query (K=8):

| Source | Count | Role | |---|---|---| | Expert demonstration | 1 | Anchors correctness | | Teacher distillation | 3 | Diverse reasoning paradigms | | Self-generated | 4 | On-policy feedback, correcting internal errors |

Normalized advantage:

A(y_k) = \frac{R(y_k) - \mu(\mathcal{G}_x)}{\sigma_R(\mathcal{G}_x) + \epsilon}

The contrast: traditional SFT says "this trajectory is correct, copy it" (absolute, sparse, single); GAL says "how much better/worse is this than the group average" (relative, dense, contrastive). GAL doesn't remove expert data—it just doesn't rely on expert data alone.

2. DCR (Dynamic Coefficient Rectification)

The raw weight $1/\pi_t$ is unbounded as $\pi_t \to 0$ . DCR adaptively clips:

C(\pi_t) = \begin{cases} \text{sg}(\pi_t) & \text{if } \pi_t < \tau \\ 1 & \text{if } \pi_t \geq \tau \end{cases}

For low-confidence tokens ( $\pi_t < 0.7$ ), the effective coefficient becomes $C(\pi)/\pi \approx 1$ (a constant), eliminating gradient explosion while preserving the learning signal.

3. Unified objective

\nabla_\theta \mathcal{L} = \mathbb{E}_{y_k \in \mathcal{G}_x} \left[ A(y_k) \frac{C(\pi_\theta(y_k|x))}{\pi_\theta(y_k|x)} \nabla_\theta \log \pi_\theta(y_k|x) \right]

Combining standardized advantages (GAL), stabilized weights (DCR), and the standard policy gradient.

Experiments

Main results (Qwen2.5-Math-1.5B)

| Method | AMC23 | College Math | MATH | Minerva | TabMWP | |---|---|---|---|---|---| | Base | 30.16 | 24.30 | 46.54 | 10.51 | 24.55 | | +SFT (100k samples) | 31.25 | 36.45 | 60.66 | 23.99 | 79.34 | | +GRPO | 44.84 | 35.58 | 65.97 | 21.17 | 76.94 | | +DFT | 36.40 | 38.76 | 64.35 | 23.75 | 82.08 | | +GFT (10k queries = 80k samples) | 46.09 | 40.51 | 70.50 | 28.93 | 85.24 |

Key findings:

GFT beats 100k-sample SFT with 1/10 the data.
GFT even surpasses GRPO (46.09 vs 44.84 on AMC23).
The mixed data is not the main driver: GFT(no mix) ≈ GFT, SFT(mix) ≈ SFT — the gains come from the mechanism.

Ablations

| Variant | AMC23 | MATH | Olympiad | |---|---|---|---| | Remove GAL + DCR (= SFT) | 31.25 | 60.66 | 24.58 | | DCR only | 35.78 | 63.91 | 26.63 | | GAL only | 42.81 | 65.97 | 27.82 | | Full GFT | 46.09 | 70.50 | 30.52 |

Both components contribute, with a synergistic effect when combined.

RL compatibility (most striking result)

SFT → GRPO: mediocre synergy (the "synergy dilemma")
GFT → GRPO: better — GAL preserves exploration space
SFT → GFT → GRPO: best performance ceiling

GFT doesn't replace SFT; it serves as a bridge from SFT to RL: SFT provides reliable initialization and format alignment, GFT restores exploration and prevents distribution drift, and GRPO exploits high-quality trajectories to reach the ceiling.

Catastrophic forgetting (LLaMA-3.2-3B)

| Method | MAWPS | SVAMP | MMLU-STEM | |---|---|---|---| | Base | 96.06 | 86.36 | 41.03 | | +SFT | 91.97 (-4.09) | 78.73 (-7.63) | 35.05 (-5.98) | | +GRPO | 94.60 (-1.46) | 88.11 (+1.75) | 39.48 (-1.55) | | +GFT | 95.79 (-0.27) | 84.65 (-1.71) | 43.89 (+2.86) |

GFT barely forgets and even improves on MMLU-STEM. KL-divergence analysis confirms GFT's KL is close to GRPO's and far below SFT's.

Why GFT Works

1. Repositioning SFT: redefining SFT as a degenerate RL form makes both diagnosis and treatment clear — the fix is not to discard SFT but to repair its optimization. 2. The contrast effect of group learning: like a student who truly understands by seeing multiple solutions (right and wrong) rather than memorizing one answer key; teacher-distilled samples break the illusion that there is only one correct way to write an answer. 3. DCR's sense of boundaries: confident tokens get normal learning; uncertain tokens get a helping hand without the gradient explosion. The threshold $\tau \approx 0.7$ was empirically validated. 4. Relation to SFT variants: DFT uses diverse responses but no group contrast or gradient rectification; ASFT uses prototypes without standardized advantages; GRPO is pure RL without expert anchoring. GFT unifies imitation and reinforcement.

Limitations

1. Group construction cost: K=8 responses per query means ~8× the inference cost of SFT, even though total data volume drops 10×. 2. Teacher dependence: biased or erroneous teacher outputs propagate into the group. 3. Task generality: validated mainly on mathematical reasoning; effectiveness on creative writing, multi-turn dialogue, and code generation is untested. 4. Threshold generalization: $\tau = 0.7$ is optimal for math; adaptive learning of $\tau$ is future work.

References

Paper: arXiv:2604.14258
Baselines: SFT/SFT(mix), DFT (Distillation Feedback Tuning), ASFT (prototype-based SFT), GRPO (Group Relative Policy Optimization)
Dataset: NuminaMath CoT (high school through IMO level)

Core takeaway: SFT shouldn't be abandoned — it should be upgraded. Through group contrast and gradient stabilization, a unified framework can capture the advantages of both imitation and reinforcement learning, rethinking how we teach AI to learn.