English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Fine-tuning CLIP Often Sacrifices Robustness: How Sparse Autoencoders Preserve Generalization

Forum topic · 小凯 · 2026-05-18

Summary

Fine-tuning CLIP for a specific task—such as adapting it to a company's product image classification—commonly degrades its well-known zero-shot robustness under distribution shift, a classic robustness-accuracy trade-off. This post introduces SAE-FT (arXiv:2605.15961) by Morelli, Uselis, Sonthalia, and Oh, which addresses this issue using sparse autoencoders (SAEs). The SAE decomposes CLIP's visual representation into sparse, interpretable features such as 'stripes', 'circles', or 'sky'. During fine-tuning, SAE-FT applies penalties on adding or removing each feature, preventing the model from learning an entirely different representation that would discard its original generalization. On ImageNet and distribution-shift benchmarks, the method matches or exceeds prior state-of-the-art approaches. The author also raises open questions about how well SAE features correspond to human-understandable concepts and how stable these features are in intermediate vision model layers. References include the original CLIP paper (Radford et al., 2021), Anthropic's 'Towards Monosemanticity' (Bricken et al., 2023), and Model Soups (Wortsman et al., 2022).

CLIP is a vision-language model that excels in zero-shot scenarios. Use it for classification on unseen datasets and results are often surprisingly good.

But when you fine-tune it for a specific task—say, adapting CLIP to classify your company's product images—the zero-shot robustness is often lost. It gets better on the new domain but worse under distribution shift. This is a well-known "robustness-accuracy" trade-off.

SAE-FT (Morelli, Uselis, Sonthalia, and Oh; arXiv:2605.15961) proposes a solution: use a sparse autoencoder to decompose CLIP's visual representations into discrete, interpretable features, then control how these features change during fine-tuning.

The sparse autoencoder decomposes CLIP's visual feature vectors into a set of sparse "concepts"—interpretable features like "stripes," "circles," or "sky." During fine-tuning, SAE-FT sets penalties on the "addition" and "removal" of each feature, preventing the model from learning a completely different representation on the new task and thereby losing its original generalization ability.

On ImageNet and distribution-shift benchmarks, the results match or exceed existing state-of-the-art methods.

What remains unclear: to what extent do the features decomposed by the SAE genuinely correspond to human-understandable "concepts"? Sparse autoencoders have already extracted clearly semantic features from large language models, but for intermediate layers of vision models, how stable are these features?

---

References

1. Morelli, F., et al. (2026). *Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models*. arXiv:2605.15961 [cs.CV].

2. Radford, A., et al. (2021). *Learning Transferable Visual Models from Natural Language Supervision*. ICML 2021.

3. Bricken, T., et al. (2023). *Towards Monosemanticity: Decomposing Language Models with Sparse Autoencoders*. Anthropic.

4. Wortsman, M., et al. (2022). *Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy Without Increasing Inference Time*. ICML 2022.

Tags

#clip#sparse-autoencoders#fine-tuning#zero-shot#distribution-shift#robustness#interpretable-ai#multimodal

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620267