Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual ASR in Speech LLMs

Forum topic · 小凯 · 2026-10-27

Summary

This paper investigates how to expand multilingual ASR systems to new languages without replaying source-language training data, using connector-based Speech LLMs where a trainable connector maps speech representations into a frozen large language model. The authors evaluate target-update construction and merging strategies for adapting the connector while preserving performance on previously supported languages. In single-target expansion, target-only fine-tuning and standard TIES merging learn new languages but cause severe source-language forgetting, while SSU-Wanda combined with TIES yields the best source-preserving trade-off. In multi-target expansion, update merging provides stable source retention, whereas recursive expansion can improve target adaptation under favorable ordering but is highly order-sensitive. The results show that multilingual ASR expansion requires balancing target language acquisition, source language retention, and interactions among updates for different target languages. Paper: arXiv:2610.35846.

Paper Overview

Field: NLP Authors: Yuchen Zhang, Haralambos Mouratidis, Ravi Shekhar Published: 2026-10-26 arXiv: 2610.35846

Abstract

Expanding multilingual ASR systems to new languages is challenging when adaptation must preserve previously supported languages without replaying source-language training data. We study this problem in connector-based Speech LLMs, where a trainable connector maps speech representations into a frozen large language model. We evaluate target-update construction and merging strategies for adapting the connector while preserving source-language performance. In single-target expansion, target-only fine-tuning and standard TIES merging learn new languages but cause severe source-language forgetting, while SSU-Wanda+TIES gives the best source-preserving trade-off. In multi-target expansion, update merging provides stable source retention, whereas recursive expansion can improve target adaptation under favorable orders but is highly order-sensitive. These results show that multilingual ASR expansion requires balancing target acquisition, source retention, and interactions among target-language updates.

Key Takeaways

Setting: Connector-based Speech LLMs, where only the connector is trainable while the LLM backbone stays frozen
Single-target expansion: Target-only fine-tuning and standard TIES merging acquire new languages but suffer severe forgetting of source languages
Best trade-off: SSU-Wanda+TIES offers the strongest source-preserving performance in single-target settings
Multi-target expansion: Update merging keeps source-language performance stable; recursive expansion helps under favorable language orders but is highly order-sensitive
Conclusion: Successful multilingual ASR expansion must balance new-language acquisition, source retention, and interactions among target-language updates

---

*Automatically collected on 2026-10-28.*