Two LLMs Scoring Zero Alone Combined to Score 84.3%: DeepMind's CALM Explained

Forum topic · 小凯 · 2026-10-27

Summary

A post on zhichai.net discusses the ICLR 2024 paper "LLM Augmented LLMs: Expanding Capabilities through Composition" (CALM) from Google DeepMind and the Indian Institute of Science. In the paper's synthetic key-value arithmetic task, two GPT-2 models each trained on only one sub-task scored 0.7% and 0% alone on the combined test, but reached 84.3% once connected via CALM. The method freezes both models and trains only a lightweight bridge module with cross-attention, representation projection, and gated residual connections initialized at zero, so the combined model starts identical to the base model. On real models, pairing Llama2-7B (English) with MaLA-500 (low-resource languages) improved low-resource task performance by up to 13 absolute percentage points without losing English ability. The post also covers limitations: results are task-specific, mild regression on old capabilities remains, and recursive composition is unresolved. Paper and code are public.

Two Models Scoring Zero Alone, 84.3% Together: How CALM Combines LLMs

Two language models faced the same comprehensive test individually: one scored 0.7%, the other scored 0%. Connected together and re-tested: 84.3%.

This is not teamwork propaganda — it is a real experiment from the ICLR 2024 paper *LLM Augmented LLMs: Expanding Capabilities through Composition*, from Google DeepMind and the Indian Institute of Science. The method is called CALM — Composition to Augment Language Models.

An honest caveat first: the 84.3% comes from a synthetic key-value arithmetic task designed by the authors, not a measure of general capability. But precisely because it is a controlled synthetic task, it exposes the mechanism of why composition works.

Half the Skills, One Comprehensive Exam

The synthetic setup: two GPT-2 models, each trained on a single skill — model A learned key-value reasoning task V1, model B learned task V2, and neither saw the full multi-step task. Then a combined exam V3 chains both abilities together.

Results: A alone scored 0.7%, B alone 0% — both useless. But connected via CALM, on the same exam: 84.3%.

Each model brought half a bucket of water; why did pouring them together fill a whole bucket?

The answer lies in what "useless" precisely means. A isn't dumb — its skill just isn't on the exam; same for B. Their two "half answers" are each incomplete, but the gaps don't overlap — what A lacks is exactly what B has. Composition lets A consult B's "notes" at every step of reasoning.

Three Old Roads and a Fourth

There are three conventional ways to give a model new abilities:

1. Train an all-capable model from scratch — most thorough, most expensive. 2. Fine-tune the existing model — cheaper, but old skills often degrade (the classic catastrophic-forgetting problem). 3. Weight merging — mathematically elegant, but requires the models to share architecture and origin; a 7B Llama and a small model from another lab simply don't fit together.

CALM takes a fourth path: freeze both models entirely, and train only a lightweight bridge module between them.

Concretely, the base model gets cross-attention at every layer — instead of attending within itself, each position of the base model can look across the other model's full internal representations and draw on relevant signals. Since the two models' internal representation "languages" differ, the bridge includes projection layers for translation. The translated signal doesn't overwrite the base model's computation; it's added via a gated residual, where each channel has a learnable tanh gate initialized at zero. So the combined model starts exactly equal to the original base model, then opens the gates gradually during training.

The design is like not re-training two experts, but simply running an intercom line between their offices. The line is new; both experts' skills remain untouched.

Does It Work on Real Models?

Synthetic tasks are clean but toys. The paper ran the same experiments on real models.

The most striking result: base model Llama2-7B, solid in English but nearly useless in low-resource languages, paired with MaLA-500, which covers a dozen low-resource languages but has poor English. Connected, low-resource language tasks improved by up to 13 absolute percentage points, with no loss in English. Experiments on math reasoning and code showed improvements in the same direction — new branches grew beyond existing strengths while strengths stayed put.

That is what the "frozen weights" conservatism buys: old capabilities are structurally guaranteed, not slowly eroded by training.

Gates Behind the Gates

Boundaries must be stated clearly, and both the post and the paper acknowledge them:

The 84.3% is task-specific and cannot be extrapolated to "composed models are universally stronger." Real-world skill overlap is far messier.
Mild regression on old capabilities remains on some tasks; gating is a safety net, not a guarantee.
Recursive composition — what happens when a third or fourth model plugs in, with information passing through successive translations — is left unresolved.

In the bigger picture, model composition has four routes: weight merging operates in parameter space; MoE trains experts together from the start with routing at inference; tool calling chains models at the output layer; CALM bridges at the internal representation layer. CALM's position: it lowers the cost of acquiring new capability from retraining a whole model to training an interface.

"Keep scaling" vs. "move to composition" was never either/or. Scale determines how deep each model's half-bucket is; composition determines whether the water can be poured together. CALM shows that the pouring can be done with just an intercom line.

Paper and code are public: arxiv.org/abs/2401.02412, github.com/google-deepmind/calm.

*Source: zhichai.net forum post (in Chinese).*