MMDiff: Multimodal Model Diffing for Feature Discovery and Control in Multimodal LLMs

Forum topic · 小凯 · 2026-08-11

Summary

Researchers introduce MMDiff, a multimodal model-diffing framework that turns sparse autoencoders (SAEs) into feature-level interfaces for auditing and controlling multimodal large language models (MLLMs). MMDiff supports three uses: isolating features altered by multimodal training by diffing a base-LM SAE against its multimodal-adapted counterpart, task-specific feature detection via per-token contrastive trigger analysis, and feature-level control through causal ablation or steering of discovered directions. The authors train multimodal SAEs for three MLLM families (LLaVA-MORE, PaliGemma 2, and InternVL3.5) and evaluate on visual spatial understanding, multimodal safety, and OCR. MMDiff finds sparse, task-specific features whose ablation selectively degrades targeted behavior by 12% on spatial tasks and 17% on OCR, and reduces multimodal safety attack success rate by 24% without hurting VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over standard single-layer steering baselines. Results show multimodal SAEs can serve as mechanisms for auditing and controlling MLLM behavior. Paper: arXiv 2508.03805.

Paper Overview

Field: NLP Authors: Hunar Batra, Lachin Naghashyar, Ashkan Khakzar Published: 2026-08-11 arXiv: 2508.03805

Abstract

Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control.

The authors introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses:

1. Feature isolation — diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training. 2. Task-specific feature detection — per-token contrastive trigger analysis to isolate causal features. 3. Feature-level control — causal ablation or steering of discovered feature directions.

Evaluation and Results

Multimodal SAEs were trained for three MLLM families (LLaVA-MORE, PaliGemma 2, and InternVL3.5) and evaluated on visual spatial understanding, multimodal safety, and OCR. Key findings:

MMDiff discovers sparse, task-specific features whose removal selectively degrades targeted behavior by 12% on spatial tasks and 17% on OCR, and reduces multimodal safety attack success rate by 24%, without affecting VQA performance.
Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average compared to standard single-layer steering baselines.

These results demonstrate that multimodal SAEs can serve not only as interpretability tools but also as mechanisms for auditing, steering, and controlling MLLM behavior toward safer and more capable generation.

---

*Auto-collected on 2026-08-12*