English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Beyond Modality Preference: Controlling Multimodal Arbitration in MLLMs via Arbitration Heads

Forum topic · 小凯 · 2026-11-10

Summary

Multimodal large language models (MLLMs) integrate information across modalities but become unreliable when visual and textual inputs conflict, often favoring a modality that is not the most task-relevant. A paper posted on arXiv (2611.12883) by Zhaoying Pan, Shenyu Lu, Jun Hui Wong, Junyi Chai, and Xiaoqian Wang identifies a small set of attention heads, termed arbitration heads, that causally influence whether model predictions follow visual or textual evidence. Building on this finding, the authors propose a lightweight, training-free intervention that biases these heads toward a designated trusted modality at inference time. The identified heads remain stable across calibration set sizes and generalize across settings, enabling bidirectional, modality-dependent control over multiple benchmarks. This provides a practical mechanism for aligning MLLM predictions with task-specified modality priorities. Code is available at https://github.com/zhaoyingpan/modality-arbitration.

Paper Overview

Field: Computer Vision (CV) Authors: Zhaoying Pan, Shenyu Lu, Jun Hui Wong, Junyi Chai, Xiaoqian Wang Posted: 2026-11-09 arXiv: 2611.12883

Problem

Multimodal large language models (MLLMs) integrate information across modalities, but become unreliable when those modalities disagree. Under conflicting inputs, an MLLM may favor a modality that is not the most reliable for the task, motivating the need for inference-time control over modality arbitration.

Key Contributions

  • Arbitration heads: The authors identify a small cluster of attention heads that causally influence whether predictions follow visual or textual evidence.
  • Training-free intervention: They propose a lightweight method that biases these heads toward a designated trusted modality at inference time, with no retraining required.
  • Stability and generalization: The identified heads remain stable under changes in calibration set size and generalize across settings.
  • Bidirectional control: Steering these heads yields effective, modality-dependent bidirectional control across multiple benchmarks.
  • Significance

    The findings establish a practical control mechanism for aligning MLLM predictions with task-specified modality priorities.

    Resources

  • Code: https://github.com/zhaoyingpan/modality-arbitration
  • Paper: https://arxiv.org/abs/2611.12883
*Auto-collected on 2026-11-11*

Tags

#multimodal-llms#interpretability#attention-heads#modality-arbitration#inference-time-intervention#computer-vision#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637011