English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

MMDiff: Multimodal Model Diffing for Feature Discovery and Control

Forum topic · 小凯 · 2026-08-11

Summary

MMDiff is a multimodal model-diffing framework that trains multimodal sparse autoencoders (SAEs) and turns them into feature-level interfaces for discovering and controlling multimodal behavior in Multimodal Large Language Models (MLLMs). The framework supports three uses: (i) feature isolation by diffing a base-LM SAE against its multimodal-adapted counterpart, (ii) task-specific feature detection via per-token contrastive trigger analysis, and (iii) feature-level control through causal ablation or steering of discovered feature directions. The authors trained multimodal SAEs for three MLLM families (LLaVA-MORE, PaliGemma 2, and InternVL3.5) and evaluated on visual spatial understanding, multimodal safety, and OCR. MMDiff identified sparse, task-specific features whose ablation selectively reduced targeted behavior by 12% on spatial tasks, 17% on OCR, and 24% attack success rate on multimodal safety attacks without hurting VQA performance. Steering these features improved spatial and OCR accuracy by +3.6% and +1.8% on average over standard single-layer steering baselines. Results show multimodal SAEs can serve not only as interpretability tools but as mechanisms for auditing, steering, and controlling MLLM behavior. Paper: arXiv 2508.03805.

Paper Overview

Field: NLP Authors: Hunar Batra, Lachin Naghashyar, Ashkan Khakzar Published: 2026-08-11 arXiv: 2508.03805

Summary

Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While hidden states decomposed into interpretable feature directions using sparse autoencoders (SAEs) are applicable to post-hoc inspection, they neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control.

The authors introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses:

1. Feature isolation — diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training. 2. Task-specific feature detection — per-token contrastive trigger analysis to isolate causal features. 3. Feature-level control — causal ablation or steering of the discovered feature directions.

Experiments and Results

Multimodal SAEs were trained for three MLLM families — LLaVA-MORE, PaliGemma 2, and InternVL3.5 — and evaluated on visual spatial understanding, multimodal safety, and OCR.

Key findings:

  • MMDiff discovers sparse, task-specific features whose ablation selectively reduces targeted behavior:
  • Spatial tasks: −12%
  • OCR: −17%
  • Multimodal safety attacks: attack success rate reduced by 24%, with no impact on VQA performance
  • Steering the discovered features improves accuracy over standard single-layer steering baselines by +3.6% on spatial tasks and +1.8% on OCR, on average.

Conclusion

These results indicate that multimodal SAEs can serve not only as interpretability tools but also as mechanisms for auditing, steering, and controlling MLLM behavior toward safer, more capable generation.

Original paper: arXiv:2508.03805

Tags

#mmdiff#sparse-autoencoders#multimodal-llms#interpretability#model-diffing#ai-safety#feature-steering#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633357