Summary
This paper introduces MMDG-Bench, the first unified and comprehensive benchmark for multimodal domain generalization (MMDG). The authors, Hao Dong, Hongzhao Li, Shupan Li, Muhammad Haris Khan et al., address the field's fragmentation, where inconsistent datasets, modality configurations, and experimental protocols make it unclear whether reported gains reflect real algorithmic progress. MMDG-Bench standardizes evaluation across six datasets and three tasks—action recognition, mechanical fault diagnosis, and sentiment analysis—covering six modality combinations, nine representative methods, and multiple evaluation settings. Beyond standard accuracy, it tests corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection. From 7,402 trained neural networks across 95 unique cross-domain tasks, five key findings emerge: (1) recent specialized MMDG methods offer only marginal improvements over the ERM baseline under fair comparison; (2) no single method consistently outperforms others across datasets or modality combinations; (3) a significant gap to upper-bound performance shows MMDG remains far from solved; (4) three-modality fusion does not consistently beat the strongest two-modality configuration; (5) all evaluated methods degrade substantially under corruption and missing modality, with some further harming model trustworthiness. Paper: arXiv 2605.06643.
Paper Overview
- Research area: Computer Vision (CV)
- Authors: Hao Dong, Hongzhao Li, Shupan Li, Muhammad Haris Khan et al.
- arXiv: 2605.06643
Background
Although multimodal domain generalization (MMDG) is increasingly popular for improving model robustness, it remains unclear whether the reported performance gains reflect genuine algorithmic progress or artifacts of inconsistent evaluation protocols. Current research is highly fragmented: studies differ significantly in datasets, modality configurations, and experimental setups. Moreover, existing benchmarks focus mainly on action recognition and often overlook critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness. This lack of standardization obscures reliable assessment of progress in the field.MMDG-Bench
To address this, the authors introduce MMDG-Bench, the first unified and comprehensive MMDG benchmark:
- Standardized evaluation on six datasets spanning three tasks: action recognition, mechanical fault diagnosis, and sentiment analysis
- Six modality combinations and nine representative methods
- Multiple evaluation settings beyond standard accuracy: corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection
- A total of 7,402 trained neural networks across 95 unique cross-domain tasks
Five Key Findings
1. Under fair comparison, recent specialized MMDG methods provide only marginal improvements over the ERM baseline.
2. No single method consistently outperforms the others across datasets or modality combinations.
3. A significant gap remains to upper-bound performance, indicating MMDG is far from solved.
4. Three-modality fusion does not consistently outperform the strongest two-modality configuration.
5. All evaluated methods degrade significantly under corruption and missing-modality scenarios, and some methods further harm model trustworthiness.Link
- arXiv: https://arxiv.org/abs/2605.06643
*Auto-collected on 2026-05-10*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619700