Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

Forum topic · 小凯 · 2026-06-26

Summary

This post introduces Facet-Probe, a five-facet audit framework (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) designed to test order invariance in multimodal large language models (MLLMs). Authored by Akshay Paruchuri, Sanmi Koyejo, and Ehsan Adeli (arXiv:2606.19224), the study audits 18 frontier and open-weight MLLMs using a Bayesian item-response model that separates ordering noise from per-facet bias, with a same-ordering control estimating the decoder-stochastic floor for observed answer flips. Key findings: none of the 18 audited MLLMs are order-invariant; screened per-facet panel-mean flip rates span 24-50%; a Gemini same-ordering control at temperature 0 shows substantial ordering excess beyond decoder noise; capability reduces but does not eliminate flipping, with the best model still flipping on 13.4% of trials. Training-free prompt-based mitigations in Gemini tests proved modality-conditional and failed to transfer from text to visual reasoning. The authors propose cross-ordering flip rate as a standard reporting axis for MLLM evaluation.

Overview

Field: NLP Authors: Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli Published: 2026-06-25 arXiv: 2606.19224

Abstract

Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer — a baseline reliability property called for by emerging AI evaluation guidelines.

The authors introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips.

Key findings

None of the 18 audited MLLMs are order-invariant: screened per-facet panel-mean flip rates span 24–50%.
A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess relative to the same-input decoder noise floor.
Capability predicts but does not eliminate flipping: the best model still flips in 13.4% of trials.
Training-free prompt-based mitigations are modality-conditional: in Gemini mitigation tests, they do not transfer from text to visual reasoning.

Implications

These results suggest prompt-level mitigation alone is unlikely to deliver general order robustness, motivating future training-time and architectural approaches. The authors propose cross-ordering flip rate as a standard reporting axis for MLLM evaluation.

--- *Auto-collected on 2026-06-26*