[论文] EI: Early Intervention for Multimodal Imaging based Disease Recognitio...

## 论文概要 **研究领域**: CV **作者**: Qijie Wei, Hailan Lin, Xirong Li **发布时间**: 2025-03-18 **arXiv**: [2503.13833](https://arxiv.org/abs/2503.13833) ## 中文摘要 当前基于多模态医学影像的疾病识别方法面临两大挑战。首先，主流的"单模态图像嵌入后融合"范式无法充分利用多模态数据中的互补和相关信息。其次，标记的多模态医学图像稀缺，加上它们与自然图像的显著领域偏移，阻碍了尖端视觉基础模型（VFMs）在医学图像嵌入中的使用。为联合解决这些挑战，我们提出了一种新颖的早期干预（EI）框架。将一个模态作为目标，其余作为参考，EI利用参考中的高级语义令牌作为干预令牌，在早期阶段引导目标模态的嵌入过程。此外，我们引入了低变化秩混合自适应（MoR），一种参数高效的微调方法，采用一组具有变化秩的低秩适配器和权重松弛路由器进行VFM自适应。在三个公共数据集（视网膜疾病、皮肤病变和膝关节异常分类）上的大量实验验证了所提方法相对于多种竞争基线的有效性。 ## 原文摘要 Current methods for multimodal medical imaging based disease recognition face two major challenges. First, the prevailing "fusion after unimodal image embedding" paradigm cannot fully leverage the complementary and correlated information in the multimodal data. Second, the scarcity of labeled multimodal medical images, coupled with their significant domain shift from natural images, hinders the use of cutting-edge Vision Foundation Models (VFMs) for medical image embedding. To jointly address the challenges, we propose a novel Early Intervention (EI) framework. Treating one modality as target and the rest as reference, EI harnesses high-level semantic tokens from the reference as intervention tokens to steer the target modality's embedding process at an early stage. Furthermore, we introdu... --- *自动采集于 2026-03-19* #论文 #arXiv #CV #小凯