CLIP is a vision-language model that excels in zero-shot scenarios. Use it for classification on unseen datasets and results are often surprisingly good.
But when you fine-tune it for a specific task—say, adapting CLIP to classify your company's product images—the zero-shot robustness is often lost. It gets better on the new domain but worse under distribution shift. This is a well-known "robustness-accuracy" trade-off.
SAE-FT (Morelli, Uselis, Sonthalia, and Oh; arXiv:2605.15961) proposes a solution: use a sparse autoencoder to decompose CLIP's visual representations into discrete, interpretable features, then control how these features change during fine-tuning.
The sparse autoencoder decomposes CLIP's visual feature vectors into a set of sparse "concepts"—interpretable features like "stripes," "circles," or "sky." During fine-tuning, SAE-FT sets penalties on the "addition" and "removal" of each feature, preventing the model from learning a completely different representation on the new task and thereby losing its original generalization ability.
On ImageNet and distribution-shift benchmarks, the results match or exceed existing state-of-the-art methods.
What remains unclear: to what extent do the features decomposed by the SAE genuinely correspond to human-understandable "concepts"? Sparse autoencoders have already extracted clearly semantic features from large language models, but for intermediate layers of vision models, how stable are these features?
---
References
1. Morelli, F., et al. (2026). *Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models*. arXiv:2605.15961 [cs.CV].
2. Radford, A., et al. (2021). *Learning Transferable Visual Models from Natural Language Supervision*. ICML 2021.
3. Bricken, T., et al. (2023). *Towards Monosemanticity: Decomposing Language Models with Sparse Autoencoders*. Anthropic.
4. Wortsman, M., et al. (2022). *Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy Without Increasing Inference Time*. ICML 2022.