Supervised fine-tuning (SFT) is highly effective on small-scale deep neural networks, but on large language models (LLMs) it sometimes produces inconsistent or even harmful results. Zhang, Cheng, Zhang, et al. investigated why from the perspective of interactive interpretability — using token-to-token interactions to quantify the reasoning patterns encoded by an LLM.
Two findings explain the apparent contradiction around SFT:
1. SFT is mainly denoising, not acquiring new knowledge. It removes noisy, unreliable interactions — cleaning up spurious token associations from the pretrained model — but rarely establishes reliable new interactions. In other words, SFT does not teach the model much genuinely new knowledge.
2. The denoising phase is extremely short. After only a few fine-tuning steps, continuing training starts introducing *overfitted interactions*, where the model learns spurious patterns in the training data rather than genuine reasoning rules.
This provides a new theoretical justification for early stopping during LLM fine-tuning, and aligns with empirical observations such as LIMA's "less is more" result for alignment.
Open questions
- How can we automatically detect the moment "denoising is complete"? This would require an online method for monitoring changes in interaction signals.
- Can interaction analysis help distinguish "beneficial knowledge injection" from "overfitted noise"?
- Do similar interaction-decay patterns appear in other alignment methods such as RLHF or DPO?
References
1. Zhang, J., Cheng, L., Zhang, G., et al. (2026). *Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspective*. arXiv:2605.17967 [cs.AI]. 2. Zhou, C., et al. (2024). *LIMA: Less Is More for Alignment*. NeurIPS. 3. Zhang, Q., et al. (2024). *Towards Understanding the Relationship Between In-Context Learning and Compositional Sparsity*. ICLR.