SpeechCombine: Adding Weights Instead of Instruction Tuning Makes Speech Models Follow Instructions

Forum topic · ✨步子哥 · 2026-07-03

Summary

SpeechCombine, an ICML 2026 paper from Tsinghua University, Shanghai Jiao Tong University, and Tencent AI Lab, shows that speech language models can acquire instruction-following ability without any speech instruction data, SFT, or RLHF. The method applies simple weight arithmetic: it adds the difference vector between a text LLM's base and instruct weights (QWEN3-8B) to a speech-pretrained model, using theta_combined = theta_speech + 0.85 * (theta_inst - theta_base). The speech model is trained with LoRA (rank 64) on only 30k hours of speech data via next-token prediction. Despite this minimal pipeline, SpeechCombine outperforms models like GLM-4-Voice, Audio-Flamingo, Step-Audio2-mini, QWEN-2.5-Omni, and Kimi-Audio that were trained on far larger instruction datasets. Key findings include that deep combination (adding weights across all layers) beats shallow combination, LoRA serves as a protective shell preserving speech capabilities, and long chain-of-thought reasoning behavior also transfers across modalities. The paper suggests model capabilities are abstract, transferable vectors rather than modality-bound skills, pointing to a cheaper paradigm for building multimodal models.

> Paper: Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning > Authors: Congrui Du, Ziyang Ma, Yifan Yang, Jian Yang, Xie Chen, Wei-Qiang Zhang > Affiliations: Tsinghua University, Shanghai Jiao Tong University, Tencent AI Lab > Venue: ICML 2026 > Link: https://arxiv.org/abs/2607.02214 > Code: https://github.com/CongruiDu/SpeechCombine

A Counterintuitive Finding

In the LLM era, nearly every capability follows the same recipe: collect instruction data → SFT → RLHF → capability acquired. Want coding? Feed code instruction data. Want math? Feed math data. Want the model to understand spoken instructions? Feed speech instruction data.

The SpeechCombine paper says: wait — speech instructions may not need instruction data at all.

The Tsinghua-led team found something striking: the "instruction-following ability" of a text LLM can be transferred to a speech model via simple weight arithmetic, with no speech instruction data, no SFT, no RL. One equation:

\theta_{combined} = \theta_{speech} + \lambda \cdot (\theta_{inst} - \theta_{base})

with $\lambda = 0.85$ .

That's it. But the results are remarkable.

Background: The Data Bloat Problem in Speech LLMs

Training a speech language model (SLM) that understands instructions conventionally requires:

1. Speech pretraining: next-token prediction on large speech corpora 2. Speech instruction tuning: SFT on "speech instruction → speech response" pairs 3. Preference alignment: RLHF or DPO

Step 2 is the biggest bottleneck, due to data bloat: "How are you" is 5 text tokens but potentially hundreds of speech tokens. A 10-second spoken instruction might be 20 text tokens but 500+ speech tokens — the same information costs 10–100x more training resources. Worse, speech instruction data has no natural source (unlike web text) and must be TTS-synthesized, limiting quality and diversity.

Result: current SLMs lag far behind text LLMs in instruction following. Not an algorithm problem — a data problem.

Core Idea: Capabilities Are Vectors You Can Add

SpeechCombine builds on the validated phenomenon that post-finetuning weight changes (task vectors) compose linearly. Given a base model $\theta_{base}$ and its instruction-tuned version $\theta_{inst}$ , the difference $\Delta\theta = \theta_{inst} - \theta_{base}$ encodes instruction following. Prior work (Chat Vector, BILLY, Preference Vector) validated this within text models — but nobody had tested cross-modal transfer.

SpeechCombine's recipe:

1. Take a text LLM's base and instruct versions (QWEN3-8B) 2. Speech-pretrain the base via LoRA (rank 64) to get $\theta_{speech}$ — it can continue speech but doesn't understand instructions 3. Add directly: $\theta_{combined} = \theta_{speech} + 0.85 \cdot (\theta_{inst} - \theta_{base})$

No SFT, no RL, no speech instruction data. Speech pretraining used only 30k hours in a single round of next-token prediction.

Results: Beating Brute-Force Competitors

| Model | Training | SpeechCombine vs. | |-------|----------|-------------------| | GLM-4-Voice | Large-scale speech instruction SFT | Better | | Audio-Flamingo | Instruction SFT + RL | Better | | Step-Audio2-mini | Industrial training | Better | | QWEN-2.5-Omni | Omnimodal training | Better | | Kimi-Audio | Large-scale training | Better on most benchmarks |

A model with 30k hours of speech data plus one line of weight addition beats industrial models trained on millions of hours with elaborate pipelines.

Why It Works: Two Key Design Choices

1. Deep vs. Shallow Combination

Shallow combination: adding weights only at the embedding and output layers
Deep combination: adding weights across all layers

Experiments show deep combination works far better, implying instruction following is not a surface-level trick but distributed across all layers — from low-level feature extraction to high-level reasoning.

2. LoRA's Crucial Role

Speech pretraining uses LoRA (rank 64) rather than full finetuning. This matters subtly: LoRA updates only a small low-rank subset of parameters, so $\theta_{speech}$ stays close to $\theta_{base}$ , differing mainly in speech-related parameters. Adding $\Delta\theta$ then primarily affects instruction-following parameters rather than disrupting speech processing. Full finetuning could drift $\theta_{speech}$ too far, and $\Delta\theta$ might then interfere with speech abilities. LoRA acts as a "protective shell."

Bonus: Long Chain-of-Thought Transfers Too

An unexpected finding: SpeechCombine also inherits the text LLM's long-thinking ability. QWEN3-8B-instruct exhibits extended reasoning before answering; after adding $\Delta\theta$ , SpeechCombine shows similar long chain-of-thought behavior on speech tasks — extended internal reasoning before the final spoken response.

This suggests $\Delta\theta$ encodes not just "instruction following" but the text LLM's overall behavior pattern from RLHF — reasoning style, response structure, even "thinking habits." In other words, behavioral patterns can transfer across modalities as vectors.

Implications for AI Research

1. A New Paradigm for Cross-Modal Capability Transfer

If text LLM capabilities can transfer to speech via weight arithmetic, the same may hold for images, video, and robot actions. A possible standard pipeline:

1. Train a strong text LLM (cheap, abundant data) 2. Do minimal pretraining on the target modality 3. Transfer capabilities via weight arithmetic

Orders of magnitude cheaper than from-scratch SFT + RLHF in the target modality.

2. Data Stacking Isn't the Only Path

The SLM mainstream is "pile on more data." SpeechCombine is a counterexample: sometimes a smarter transfer strategy beats more data. The "without Instruction Tuning" in the title is deliberately provocative — massive speech instruction tuning may be unnecessary.

3. Rethinking What "Capability" Is

If instruction following is representable as a weight difference that transfers across modalities, then capability is not modality-bound but an abstract, transferable vector. The model doesn't "learn speech instruction following" — it "possesses abstract instruction following, with speech as a new interface." An analogy to human cognition: once you can "follow instructions" abstractly, it doesn't matter whether they arrive as text, speech, or gestures.

Honest Assessment

Caveats worth noting:

1. Depends on text LLM quality: the ceiling is locked by $\theta_{inst}$ 's strength 2. LoRA rank 64 is empirical: no theoretical explanation given 3. $\lambda = 0.85$ is tuned: different model pairs may need different values; no automatic method 4. Only tested on QWEN3: whether Llama, Mistral, etc. work equally well needs verification 5. Speech quality underexplored: evaluation focuses on instruction following; naturalness and emotion are less assessed

Closing Thoughts

SpeechCombine evokes an aesthetic principle from physics: the beauty of a theory lies in its simplicity.

While everyone marches toward complexity — more data, bigger models, more elaborate pipelines — someone looked back and found that one line of weight addition solves what seemed to require millions of hours of data.

This doesn't mean complex methods are worthless. But before pursuing complexity, ask a more fundamental question: do you really need to be this complex?

Speech instruction following may be, at heart, a text capability that merely happens to receive input through a speech interface. Transfer the text capability, and that's enough.

Sometimes the deepest insight isn't "what to do" — it's "discovering what you don't need to do."

---

Paper: https://arxiv.org/abs/2607.02214 Code: https://github.com/CongruiDu/SpeechCombine