SpeechCombine: Adding Weights Instead of Instruction Tuning Lets Speech Models Follow Instructions

Forum topic · ✨步子哥 · 2026-07-03

Summary

SpeechCombine, an ICML 2026 paper from Tsinghua University, Shanghai Jiao Tong University, and Tencent AI Lab, shows that speech language models can acquire instruction-following ability without any speech instruction data, SFT, or RL. The method computes a task vector from a text LLM's base and instruct versions (Qwen3-8B), then adds it to a speech-pretrained model via weight arithmetic: theta_combined = theta_speech + 0.85 * (theta_inst - theta_base). Speech pretraining uses only 30k hours of data with LoRA (rank 64). Despite this simplicity, SpeechCombine outperforms models like GLM-4-Voice, Audio-Flamingo, Step-Audio2-mini, Qwen2.5-Omni, and Kimi-Audio that were trained with large-scale speech instruction data. Key findings include that deep combination (adding weights across all layers) beats shallow combination, LoRA acts as a protective shell preserving speech capabilities, and long-thinking chain-of-reasoning behavior transfers from the text LLM as well. The paper suggests abstract capabilities like instruction following are modality-agnostic, transferable vectors, offering a cheaper alternative to data-heavy speech instruction tuning.

> Paper: Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning > Authors: Congrui Du, Ziyang Ma, Yifan Yang, Jian Yang, Xie Chen, Wei-Qiang Zhang > Institutions: Tsinghua University, Shanghai Jiao Tong University, Tencent AI Lab > Venue: ICML 2026 > Paper: https://arxiv.org/abs/2607.02214 > Code: https://github.com/CongruiDu/SpeechCombine

A Counterintuitive Finding

In the LLM era, nearly every capability follows the same recipe: collect instruction data → SFT → RLHF → obtain the capability. Want it to follow spoken instructions? Feed it speech instruction data.

The SpeechCombine paper says: wait — speech instruction following might not need instruction data at all.

The Tsinghua-led team found that the "instruction-following ability" of a text LLM can be transferred to a speech model via simple weight arithmetic, with no speech instruction data, no SFT, and no RL. One formula:

\theta_{combined} = \theta_{speech} + \lambda \cdot (\theta_{inst} - \theta_{base})

where $\lambda = 0.85$ .

That's it. And the results are striking.

Background: The Data Inflation Problem in Speech LLMs

Training a speech language model (SLM) that can follow instructions normally requires:

1. Speech pretraining: next-token prediction over massive speech data to learn speech-token statistics 2. Speech instruction tuning: SFT on "speech instruction–speech response" pairs 3. Preference alignment: RLHF or DPO

Step 2 is the biggest bottleneck due to data inflation: "How are you" is 5 text tokens but may span hundreds of speech tokens. A 10-second spoken instruction might be ~20 text tokens but 500+ speech tokens — a 10–100x training-resource overhead for the same information. Speech instruction pairs also have no natural data source and must be synthesized via TTS, limiting quality and diversity. As a result, current SLMs lag far behind text LLMs in instruction following. It's not an algorithm problem — it's a data problem.

Core Idea: Capabilities Are Vectors That Can Be Added

Fine-tuned weight changes (task vectors) are known to be linearly composable in text models (Chat Vector, BILLY, Preference Vector). SpeechCombine is the first to verify this cross-modally:

1. Take a text LLM's base ( $\theta_{base}$ ) and instruct ( $\theta_{inst}$ ) versions (Qwen3-8B in the paper) 2. Speech-pretrain on $\theta_{base}$ (LoRA, rank 64) to get $\theta_{speech}$ — a model that can only continue speech, not follow instructions 3. Add: $\theta_{combined} = \theta_{speech} + 0.85 \cdot (\theta_{inst} - \theta_{base})$

No SFT, no RL, no speech instruction data. Speech pretraining used only 30k hours of data in a single round of next-token prediction.

Results: Beating Heavy-Duty Competitors

Tested on multiple benchmarks, SpeechCompare outperformed models trained on large-scale speech instruction data:

| Model | Training | vs. SpeechCombine | |-------|----------|-------------------| | GLM-4-Voice | Large-scale speech SFT | SpeechCombine better | | Audio-Flamingo | Speech SFT + RL | SpeechCombine better | | Step-Audio2-mini | Industrial-grade training | SpeechCombine better | | Qwen2.5-Omni | Omnimodal training | SpeechCombine better | | Kimi-Audio | Large-scale training | SpeechCombine mostly better |

A model with 30k hours of speech data plus one weight-addition operation beat industrial models trained on millions of hours with complex pipelines.

Why It Works: Two Key Design Choices

1. Deep vs. Shallow Combination

Shallow combination: add weights only at embedding/output layers
Deep combination: add weights at all layers

Experiments show deep combination works far better, indicating instruction-following is distributed across all layers, from low-level feature extraction to high-level reasoning.

2. LoRA's Critical Role

Speech pretraining uses LoRA (rank 64) rather than full fine-tuning. This keeps most parameters close to $\theta_{base}$ , so the added task vector mainly affects instruction-following parameters rather than disturbing speech-processing ones. Full fine-tuning would drift $\theta_{speech}$ too far, and LoRA acts as a protective shell.

Unexpected Bonus: Long-Thinking Transfers Too

SpeechCombine also inherits long chain-of-thought behavior from Qwen3-8B-instruct: the model performs extended internal reasoning before producing its spoken answer. This suggests $\Delta\theta$ encodes not just instruction following but the text LLM's overall behavioral patterns — reasoning style, response structure, even "thinking habits." Model behavior patterns can transfer across modalities as vectors.

Implications for AI Research

1. A new cross-modal transfer paradigm: capabilities accumulated in text LLMs may transfer to other modalities (image, video, robotics) via weight arithmetic — orders of magnitude cheaper than SFT + RLHF in the target modality. 2. Data scaling isn't the only path: smarter transfer can beat more data. 3. A philosophical reframing of capability: instruction following is an abstract, modality-agnostic vector — speech is just a new interface for it.

Honest Assessment

Caveats worth noting:

1. Depends on the text LLM's quality: the ceiling is set by $\theta_{inst}$ 2. LoRA rank choice is empirical (no theoretical explanation for why 64 works best) 3. $\lambda = 0.85$ was tuned; no automatic method exists 4. Only tested on Qwen3; validation on Llama/Mistral architectures is needed 5. Speech quality evaluation is limited: instruction following is tested; naturalness and emotion less so

Conclusion

When everyone is scaling data, models, and pipelines, SpeechCombine shows a single weight addition can solve a problem thought to require millions of hours of data. The deepest insight is not what you do, but discovering what you don't need to do. Speech instruction following may essentially be a text capability that happens to receive input through a speech interface — transfer the text capability, and that's enough.

---

Paper: https://arxiv.org/abs/2607.02214 Code: https://github.com/CongruiDu/SpeechCombine