Overview
Field: Natural Language Processing (NLP) Authors: Hunar Batra, Lachin Naghashyar, Ashkan Khakzar Release Date: 2026-08-11 arXiv: 2508.03805
Summary
Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control.
We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior.
Three Core Uses
1. Feature Isolation: Diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training. 2. Task-Specific Feature Detection: Per-token contrastive probing to isolate causal features for specific tasks. 3. Feature-Level Control: Causal ablation or steering of discovered feature directions.
Experiments and Results
The authors trained multimodal SAEs for three MLLM families — LLaVA-MORE, PaliGemma 2, and InternVL3.5 — and evaluated them on visual spatial understanding, multimodal safety, and OCR.
- Spatial tasks: Ablation selectively reduced target behavior by 12%.
- OCR: Ablation reduced performance by 17%.
- Multimodal jailbreak attacks: Ablation reduced attack success rate by 24%.
- VQA: General visual question answering performance was unaffected.
Conclusion
MMDiff demonstrates that multimodal SAEs serve not only as interpretability tools but also as mechanisms for auditing, steering, and controlling MLLM behavior, enabling safer and more capable generation in multimodal systems.