ProtoAda: Prototype-Guided Adaptive Adapter Expansion for Multimodal Continual Instruction Tuning

Forum topic · 小凯 · 2026-06-03

Summary

ProtoAda is a prototype-guided adaptive fine-tuning framework for Multimodal Continual Instruction Tuning (MCIT), proposed by Yu-Cheng Shi, Zhen-Hao Xie, and Jun-Tao Tang (arXiv:2506.00004). The paper addresses a key limitation of sparse architectures such as Mixture of LoRA Experts with image-text similarity routing: tasks with distinct response structures can share highly similar visual-linguistic semantics and get incorrectly routed to the same expert. For example, a grounding expert that predicts coordinates may be biased toward producing short textual answers after learning semantically similar VQA tasks. This format-blind task assignment merges heterogeneous response types into shared parameters, causing gradient interference and ineffective expert collaboration. ProtoAda introduces format-aware task prototypes that align task assignment and routing with both task semantics and output structure, and geometrically integrates format-compatible updates to reuse and progressively refine existing parameters. Experiments across multiple benchmarks show superior performance, especially on tasks whose answer structures are easily disrupted by sequential fine-tuning.

Overview

Field: Computer Vision
Authors: Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang
Published: 2026-06-03
arXiv: 2506.00004

Abstract

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. To reduce inter-task interference and promote collaboration, recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing. However, tasks with distinct response structures could share highly similar visual-linguistic semantics and thus be wrongly routed to the same expert; image-text similarity alone is insufficient for reliable task assignment. For example, an expert in a grounding task requiring coordinate prediction may be biased toward producing short textual answers after learning semantically similar VQA tasks. This format-blind task assignment merges heterogeneous response types into shared parameters, leading to gradient interference and ineffective expert collaboration.

Method

To address this, the authors propose ProtoAda, a prototype-guided adaptive fine-tuning framework:

Format-aware task prototypes: task assignment and routing are aligned with both task semantics and output structure, not just image-text similarity.
Geometric-aware integration: format-compatible updates are merged in a geometry-aware manner, enabling effective reuse and progressive refinement of existing parameters.

Results

Extensive experiments across multiple benchmarks demonstrate that ProtoAda achieves superior performance, particularly on tasks whose answer structures are easily disrupted by sequential fine-tuning.