CLIP is a vision-language model that excels in zero-shot settings. When you use it for classification on unseen datasets, the results are often surprisingly good.
But when you fine-tune it for a specific task — say, adapting CLIP to classify your company's product images — the zero-shot robustness is often lost. It gets better on the new domain, but worse under distribution shift. This is the well-known robustness-accuracy trade-off.
SAE-FT (arXiv:2605.15961) by Morelli, Uselis, Sonthalia, and Oh proposes a solution: use a sparse autoencoder to decompose CLIP's visual representations into discrete, interpretable features, and then regulate how those features change during fine-tuning.
A sparse autoencoder breaks CLIP's visual feature vectors into a sparse set of "concepts" — interpretable features like "striped", "circular", or "sky". During fine-tuning, SAE-FT applies penalties on the addition and removal of each feature, preventing the model from learning an entirely different representation on the new task and thereby losing its original generalization ability.
On ImageNet and distribution-shift benchmarks, the results match or exceed existing state-of-the-art methods.
Unclear points: to what extent do the features extracted by the SAE actually correspond to human-understandable "concepts"? Sparse autoencoders have already been shown to extract clearly semantic features in large language models, but for intermediate layers of vision models, how stable are these features?
---
References
1. Morelli, F., et al. (2026). *Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models*. arXiv:2605.15961 [cs.CV]. 2. Radford, A., et al. (2021). *Learning Transferable Visual Models from Natural Language Supervision*. ICML 2021. 3. Bricken, T., et al. (2023). *Towards Monosemanticity: Decomposing Language Models with Sparse Autoencoders*. Anthropic. 4. Wortsman, M., et al. (2022). *Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy Without Increasing Inference Time*. ICML 2022.