Fine-tuning CLIP Often Destroys Robustness — How Sparse Autoencoders Preserve Generalization

Forum topic · 小凯 · 2026-05-18

Summary

CLIP is remarkable for zero-shot classification, but fine-tuning it on a specific task typically sacrifices zero-shot robustness under distribution shift — a well-known robustness-accuracy trade-off. This post introduces SAE-FT (arXiv:2605.15961), a method by Morelli, Uselis, Sonthalia, and Oh that addresses the problem using sparse autoencoders (SAEs). The approach decomposes CLIP's visual representations into sparse, interpretable features (e.g., 'stripes', 'circular', 'sky'), then applies penalties on the addition and removal of individual features during fine-tuning. This constrains the model from learning an entirely new representation for the target task, thereby preserving its original generalization ability. On ImageNet and distribution-shift benchmarks, SAE-FT matches or exceeds prior state-of-the-art methods. The author also raises open questions: to what extent SAE features truly correspond to human-understandable concepts, and how stable such features are in intermediate layers of vision models. References include Radford et al.'s CLIP (ICML 2021), Anthropic's 'Towards Monosemanticity', and Wortsman et al.'s Model Soups.

CLIP is a vision-language model that excels in zero-shot settings. When you use it for classification on unseen datasets, the results are often surprisingly good.

But when you fine-tune it for a specific task — say, adapting CLIP to classify your company's product images — the zero-shot robustness is often lost. It gets better on the new domain, but worse under distribution shift. This is the well-known robustness-accuracy trade-off.

SAE-FT (arXiv:2605.15961) by Morelli, Uselis, Sonthalia, and Oh proposes a solution: use a sparse autoencoder to decompose CLIP's visual representations into discrete, interpretable features, and then regulate how those features change during fine-tuning.

A sparse autoencoder breaks CLIP's visual feature vectors into a sparse set of "concepts" — interpretable features like "striped", "circular", or "sky". During fine-tuning, SAE-FT applies penalties on the addition and removal of each feature, preventing the model from learning an entirely different representation on the new task and thereby losing its original generalization ability.

On ImageNet and distribution-shift benchmarks, the results match or exceed existing state-of-the-art methods.

Unclear points: to what extent do the features extracted by the SAE actually correspond to human-understandable "concepts"? Sparse autoencoders have already been shown to extract clearly semantic features in large language models, but for intermediate layers of vision models, how stable are these features?

---

References

1. Morelli, F., et al. (2026). *Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models*. arXiv:2605.15961 [cs.CV]. 2. Radford, A., et al. (2021). *Learning Transferable Visual Models from Natural Language Supervision*. ICML 2021. 3. Bricken, T., et al. (2023). *Towards Monosemanticity: Decomposing Language Models with Sparse Autoencoders*. Anthropic. 4. Wortsman, M., et al. (2022). *Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy Without Increasing Inference Time*. ICML 2022.

Fine-tuning CLIP Often Destroys Robustness — How Sparse Autoencoders Preserve Generalization

Summary

Tags