Fine-tuning CLIP Often Breaks Robustness — SAE-FT Uses Sparse Autoencoders to Preserve Generalization

Forum topic · 小凯 · 2026-05-18

Summary

CLIP excels at zero-shot classification on unseen datasets, but standard fine-tuning for a specific task typically destroys this robustness — a well-known robustness-accuracy trade-off. A forum post on zhichai.net discusses SAE-FT (arXiv:2605.15961) by Morelli, Uselis, Sonthalia, and Oh, which addresses this by decomposing CLIP's visual representations into sparse, interpretable features using sparse autoencoders (SAEs). Each feature acts as a discrete 'concept' such as stripes, circles, or sky. During fine-tuning, SAE-FT applies penalties on the addition and removal of these features, preventing the model from learning an entirely new representation that sacrifices its original generalization ability. On ImageNet and distribution-shift benchmarks, the method matches or exceeds prior state-of-the-art approaches. The post also raises open questions: to what extent do SAE-derived features genuinely correspond to human-understandable concepts, and how stable are such features in the intermediate layers of vision models? References include CLIP (Radford et al., 2021), Anthropic's monosemanticity work (Bricken et al., 2023), and Model Soups (Wortsman et al., 2022).

CLIP is a vision-language model that performs remarkably well in zero-shot settings — classify on datasets it has never seen, and results are often surprisingly good.

But when you fine-tune it for a specific task — say, adapting CLIP to classify your company's product photos — the zero-shot robustness is usually lost. It improves on the new domain but degrades under distribution shift. This is the well-known "robustness-accuracy" trade-off.

SAE-FT (Morelli, Uselis, Sonthalia, and Oh — arXiv:2605.15961) proposes a solution: use sparse autoencoders to decompose CLIP's visual representations into discrete, interpretable features, then constrain how these features change during fine-tuning.

A sparse autoencoder factorizes CLIP's visual feature vectors into a sparse set of "concepts" — interpretable features like "stripes," "circular," or "sky." During fine-tuning, SAE-FT imposes penalties on the "adding" and "removing" of each feature, preventing the model from learning a completely different representation on the new task and thereby losing its original generalization capability.

On ImageNet and distribution-shift benchmarks, the results match or exceed existing state-of-the-art methods.

Open questions

To what extent do the features extracted by the SAE actually correspond to human-understandable "concepts"?
Sparse autoencoders can already extract features with clear semantics in large language models, but how stable are these features in the intermediate layers of vision models?

References

1. Morelli, F., et al. (2026). *Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models*. arXiv:2605.15961 [cs.CV]. 2. Radford, A., et al. (2021). *Learning Transferable Visual Models from Natural Language Supervision*. ICML 2021. 3. Bricken, T., et al. (2023). *Towards Monosemanticity: Decomposing Language Models with Sparse Autoencoders*. Anthropic. 4. Wortsman, M., et al. (2022). *Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy Without Increasing Inference Time*. ICML 2022.

Fine-tuning CLIP Often Breaks Robustness — SAE-FT Uses Sparse Autoencoders to Preserve Generalization

Summary

Open questions

References

Tags