When to Align, When to Predict: A Phase Diagram for Multimodal Learning

Forum topic · 小凯 · 2026-06-11

Summary

Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms in multimodal representation learning, yet no systematic theory explains when each succeeds or fails. A new paper by Ilay Kamai, Hugues Van Assel, Aviv Regev, Hagai B. Perets, and Randall Balestriero (arXiv:2606.11190) develops a unified linear framework to answer both questions. Under a spiked signal-plus-noise model with structured cross-modal nuisance correlation, the authors derive separation ratios for both objectives, revealing complementary failure modes: alignment fails when noise is strongly correlated across modalities, while prediction is limited by the quality of the source modality. The resulting phase diagram partitions multimodal problems into four regimes: both methods work, only CA works, only CP works, or neither works. Experiments on stereo vision, image-text pairs, and astrophysical data validate the nonlinear predictions, including cases where cross-modal training actively hurts performance. This gives practitioners, especially in scientific domains, a diagnostic tool for choosing the right multimodal strategy.

Paper Overview

Field: Machine Learning Authors: Ilay Kamai, Hugues Van Assel, Aviv Regev, Hagai B. Perets, Randall Balestriero arXiv: 2606.11190

Abstract

Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds, when each fails, and when cross-modal training helps at all -- a gap that leaves practitioners, especially in scientific domains like biomedicine or astrophysics, with heterogeneous instruments and multiple levels of organization and measurement, unable to diagnose why standard methods underperform the best single modality. We develop a unified linear framework that addresses both questions. Under a spiked signal-plus-noise model with structured cross-modal nuisance correlation, we derive separation ratios for both objectives that expose complementary failure modes: alignment whitens each modality and fails when noise is strongly correlated across modalities, while prediction is bounded by the quality of the source modality. The resulting phase diagram partitions multimodal problems into four regimes: both work, only CA works, only CP works, or neither works. Experiments on stereo vision, image-text pairs, and astrophysical data validate the nonlinear predictions, including cases where cross-modal training is harmful.

Key Takeaways

Unified theory: A single linear framework analyzes both CA and CP objectives.
Complementary failure modes: Alignment fails under strongly cross-modal-correlated noise; prediction is capped by source-modality quality.
Phase diagram: Four regimes — both succeed, only CA, only CP, neither.
Empirical validation: Stereo vision, image-text, and astrophysics experiments confirm the theory, including cases where cross-modal training hurts.