Introduction
This post summarizes the paper ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Co... (Computer Vision), authored by Yu-Cheng Shi, Zhen-Hao Xie, and Jun-Tao Tang. The paper is available on arXiv: 2506.00004.
Key Points
- Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.
- Recent methods reduce inter-task interference with sparse architectures, such as Mixture of LoRA Experts with image-text similarity routing.
- Problem: tasks with distinct response structures may share highly similar visual-linguistic semantics and get wrongly routed to the same expert; image-text similarity alone is insufficient for reliable task allocation. For example, an expert for a grounding task requiring coordinate prediction may be biased toward producing short textual answers after learning a semantically similar VQA task.
- This format-agnostic task assignment mixes heterogeneous response types into shared parameters, causing gradient interference and ineffective expert collaboration.
- Solution — ProtoAda: a prototype-guided adaptive fine-tuning framework that introduces format-aware task prototypes, aligning task allocation and routing with both task semantics and output structure.
- ProtoAda further integrates format-compatible updates in a geometric-aware manner, enabling effective reuse and progressive refinement of existing parameters.
- Extensive experiments across multiple benchmarks show superior performance, especially on tasks whose answer structures are easily corrupted by sequential fine-tuning.
- arXiv: <https://arxiv.org/abs/2506.00004>