MMDiff: Multimodal Model Diffing with Sparse Autoencoders for Feature Discovery and Control in MLLMs

Forum topic · 小凯 · 2026-08-11

Summary

This paper introduces MMDiff, a multimodal model-diffing framework that trains multimodal sparse autoencoders (SAEs) to serve as feature-level interfaces for discovering and controlling behavior in Multimodal Large Language Models (MLLMs). Existing SAEs decompose hidden states into interpretable feature directions for post-hoc inspection, but cannot isolate which features are altered by multimodal training or enable targeted control. MMDiff addresses this by diffing a base-LM SAE against its multimodal-adapted counterpart to identify changed features, using per-token contrastive probing to isolate causal task-specific features, and applying feature-level steering or ablation for control. The authors train multimodal SAEs for LLaVA-MORE, PaliGemma 2, and InternVL3.5, evaluating on visual spatial understanding, multimodal safety, and OCR. Results show sparse causal features whose removal selectively degrades target behavior by 12% on spatial tasks, 17% on OCR, and 24% on multimodal safety attacks without harming VQA. Steering improves spatial and OCR accuracy by +3.6% and +1.8% on average over single-layer baselines.

Paper Overview

Field: NLP
Authors: Hunar Batra, Lachin Naghashyar, Ashkan Khakzar
Published: 2026-08-11
arXiv: 2508.03805

Abstract

Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control.

We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses:

1. Feature isolation, by diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training. 2. Task-specific feature detection, via per-token contrastive probing to isolate causal features. 3. Feature-level control, through causal ablation or steering of the discovered feature directions.

We train multimodal SAEs for three MLLM families—LLaVA-MORE, PaliGemma 2, and InternVL3.5—and evaluate on visual spatial understanding, multimodal safety, and OCR.

Key Findings

MMDiff discovers sparse, causally specific features whose removal selectively reduces target behavior by 12% on spatial tasks, 17% on OCR, and 24% on multimodal safety attack success rate, without affecting general VQA performance.
Steering these features outperforms standard single-layer steering baselines by an average of +3.6% on spatial accuracy and +1.8% on OCR accuracy.
Results indicate that multimodal SAEs function not only as interpretability tools but also as mechanisms for auditing, steering, and controlling MLLM behavior toward safer and more capable generation.

Implications

The work positions multimodal SAEs as practical, feature-level interfaces for both mechanistic interpretability and safety-oriented control in vision-language models, enabling targeted auditing of behaviors introduced during multimodal fine-tuning.