Paper Overview
Field: Computer Vision (CV) Authors: Zhaoying Pan, Shenyu Lu, Jun Hui Wong, Junyi Chai, Xiaoqian Wang Posted: 2026-11-09 arXiv: 2611.12883
Problem
Multimodal large language models (MLLMs) integrate information across modalities, but become unreliable when those modalities disagree. Under conflicting inputs, an MLLM may favor a modality that is not the most reliable for the task, motivating the need for inference-time control over modality arbitration.
Key Contributions
- Arbitration heads: The authors identify a small cluster of attention heads that causally influence whether predictions follow visual or textual evidence.
- Training-free intervention: They propose a lightweight method that biases these heads toward a designated trusted modality at inference time, with no retraining required.
- Stability and generalization: The identified heads remain stable under changes in calibration set size and generalize across settings.
- Bidirectional control: Steering these heads yields effective, modality-dependent bidirectional control across multiple benchmarks.
- Code: https://github.com/zhaoyingpan/modality-arbitration
- Paper: https://arxiv.org/abs/2611.12883
Significance
The findings establish a practical control mechanism for aligning MLLM predictions with task-specified modality priorities.