Copy the Same, Distill the Difference: Initializing Linear Vision Transformers from Softmax ViTs

Forum topic · 小凯 · 2026-09-30

Summary

Linear Vision Transformers (ViTs) replace Softmax attention with linear-complexity operators for efficient token routing, but typically require costly from-scratch pre-training and underperform their Softmax counterparts. This paper investigates whether linear ViTs can reuse pre-trained weights from foundation Softmax ViTs. The authors find that transfer behaves oppositely to prior attention-transfer findings: attention weights are operator-specific and copying them directly is unhelpful, sometimes worse than random initialization. Instead, attention's token-routing behavior can be recovered through distillation with appropriate loss design, allowing linear ViTs to close the gap or even match Softmax models. In contrast, MLP weights, which carry learned representations, are operator-agnostic and transfer effectively via simple copying, already delivering most of the pre-training benefit. Combining copied MLPs with distilled attention enables linear ViTs to bridge or even exceed Softmax performance. Findings hold across linear ViT variants, model scales, and diverse datasets.

Paper Overview

Field: Computer Vision (CV) Authors: Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Shiqi Huang, Min Kass Chong, Wahyu Wiratama, Peng Hu, Chen Gong, Wu Liu, Xi Peng, Chun Jian Ho, Hongyuan Zhu Published: 2026-09-28 arXiv: 2609.35745

Abstract

Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear.

In this work, the authors explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights?

Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, the authors find the opposite for Softmax-to-linear transfer:

Attention weights are operator-specific: copying them directly offers almost no benefit, and can sometimes perform worse than random initialization.
Instead, the attention's token-routing behavior can be recovered via distillation with appropriate loss design, allowing linear ViTs to narrow the gap and even match their Softmax counterparts.
MLP weights, which carry learned representations, are operator-agnostic: they can be transferred effectively through simple direct copying, already yielding most of the benefit of pre-trained weights.

Therefore, copying the MLPs serves as an effective foundation for Softmax-to-linear transfer: combined with distillation-recovered attention, linear ViTs finally close the remaining gap and can even surpass the Softmax versions.

These findings are consistent across different linear ViT variants, model scales, and diverse datasets. The takeaway: copy the same parts, distill the different parts — recovering benefits across the Softmax-to-linear boundary.

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers from Softmax ViTs

Summary

Paper Overview

Abstract

Links

Tags