EI: Early Intervention Framework for Multimodal Medical Imaging Disease Recognition

Forum topic · 小凯 · 2026-03-19

Summary

This paper (arXiv:2503.13833) addresses two challenges in multimodal medical imaging-based disease recognition: the prevailing 'fusion after unimodal image embedding' paradigm fails to fully exploit complementary and correlated information across modalities, and the scarcity of labeled multimodal medical images—combined with domain shift from natural images—hinders the use of Vision Foundation Models (VFMs). The authors propose an Early Intervention (EI) framework that treats one modality as target and the rest as reference, using high-level semantic tokens from the reference to steer the target modality's embedding at an early stage. They also introduce Mixture of Low-rank Adapters with Varying Ranks (MoR), a parameter-efficient fine-tuning method with low-rank adapters of varying ranks and a weight-relaxed router for VFM adaptation. Experiments on three public datasets—retinal disease, skin lesion, and knee abnormality classification—show effectiveness over competitive baselines.

Paper Overview

Research Area: Computer Vision (CV) Authors: Qijie Wei, Hailan Lin, Xirong Li Published: 2025-03-18 arXiv: 2503.13833

English Abstract

Current methods for multimodal medical imaging based disease recognition face two major challenges. First, the prevailing "fusion after unimodal image embedding" paradigm cannot fully leverage the complementary and correlated information in the multimodal data. Second, the scarcity of labeled multimodal medical images, coupled with their significant domain shift from natural images, hinders the use of cutting-edge Vision Foundation Models (VFMs) for medical image embedding.

To jointly address these challenges, the authors propose a novel Early Intervention (EI) framework. Treating one modality as target and the rest as reference, EI harnesses high-level semantic tokens from the reference as intervention tokens to steer the target modality's embedding process at an early stage.

Additionally, they introduce MoR (Mixture of low-Rank adapters with varying ranks), a parameter-efficient fine-tuning method that uses a set of low-rank adapters with varying ranks together with a weight-relaxed router for VFM adaptation.

Extensive experiments on three public datasets—retinal disease, skin lesion, and knee abnormality classification—validate the effectiveness of the proposed method relative to a variety of competitive baselines.

Key Contributions

1. Early Intervention (EI) framework: moves cross-modal fusion earlier in the embedding pipeline, using reference-modality semantic tokens to guide target-modality embedding. 2. MoR fine-tuning: parameter-efficient VFM adaptation via varying-rank low-rank adapters with a weight-relaxed router. 3. Empirical validation on three multimodal medical imaging tasks: retinal disease, skin lesion, and knee abnormality classification.

---

*Auto-collected on 2026-03-19*