One Transformer, Two Thoughts: Linear Superposition in LLMs and Dual-Stream Decoding
A Counterintuitive Experiment
Take two completely unrelated texts—stream A about quantum computing, stream B about Italian recipes. Average their embeddings token-by-token (not concatenation, not interleaving—direct addition divided by two), and feed this mixed signal into a pretrained LLM.
Intuition says the model should break down. Transformers are full of nonlinear components: self-attention uses softmax, MLPs use GELU/SiLU activations, layer normalization is nonlinear too. Forcing two semantic streams through a single forward pass should be like playing two songs simultaneously—you'd expect noise.
But Pavel Tikhonov and colleagues ran exactly this experiment, and the result is jaw-dropping: the model didn't break—its single forward pass produced next-token predictions for both streams. The correct continuation of stream A appeared in the top-10, and so did the correct continuation of stream B. One forward pass, two predictions, coexisting.
This is the core finding of the paper *"Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs"*—the Superposition Linearity Hypothesis.
What Is "Linear Superposition"
In plain terms: a function f is linear if f(αx + βy) = αf(x) + βf(y). Linear functions have a beautiful property—linear combinations of inputs map to linear combinations of outputs, so two information channels can superimpose independently without interference.
Transformers are clearly not linear. Softmax redistributes probability, attention weights are nonlinear mappings, MLP activations are nonlinear. Mathematically, Transformers shouldn't have this property.
But the experimental data says: they do.
Specifically, when you average two text streams' embeddings positionally to get x_mix = (x_A + x_B) / 2, run one forward pass to get p_mix, you find:
- Stream A's next token retains significant probability mass in p_mix
- Stream B's next token also retains significant probability mass
- p_mix approximately equals the average of p_A and p_B
Not Learned—Baked Into the Architecture
A key question: is this linear superposition learned during training, or inherent to the Transformer architecture itself?
The authors' answer, from tracking the pretraining process: superposition is strongest at initialization and gradually weakens as pretraining progresses.
In other words, a randomly initialized Transformer has the best superposition. The more training, the worse it gets. This means linear superposition isn't a learned capability—it's the architecture's "factory setting." The training objective, optimized for single-stream language modeling, gradually erodes this property.
This aligns with another recent research thread: transitions between adjacent layers in decoder-only Transformers' residual streams are well approximated by affine maps. Layer-to-layer behavior is approximately linear, and end-to-end behavior retains a substantial degree of linearity.
0.025% of Data Restores Linearity
Pretraining damages linearity, but not irreversibly. The authors found that a very lightweight fine-tuning stage—using less than 0.025% of the original pretraining dataset size—substantially restores superposition.
That number is worth pausing on. 0.025% means: if original pretraining used 4 trillion tokens, fine-tuning needs only around 1 billion tokens. This echoes the spirit of the LIMA paper ("Less Is More for Alignment"): many capabilities are already acquired during pretraining and just need small amounts of data to "activate."
After fine-tuning, superposition improves significantly—the mixed-input output distribution becomes closer to the average of the two independent distributions, and both streams' correct continuation tokens rank higher in the top-10.
Multitasking: Decoding Two Texts from One Forward Pass
If superposition is good enough, can it be exploited in reverse? The authors propose a bold decoding scheme: generate two independent continuations simultaneously from a single mixed forward pass.
The approach:
1. Mix the embeddings of streams A and B, run one forward pass 2. Get the mixed logits distribution 3. Use "Joint Contrastive Decoding" to separate the two streams' predictions 4. Sample separately for each stream, producing two independent continuations
This is the model "multitasking"—one computation, two outputs. If it works reliably, inference throughput doubles directly.
The authors achieved it. On the fine-tuned model, both continuations decoded from the same mixed forward pass were coherent and semantically correct. One notable detail: directly using the geometric mean to separate streams runs into obstacles—the two streams' probability distributions aren't in a simple additive relationship, requiring a more refined decoding strategy. "Joint Contrastive Decoding" bypasses this obstacle.
Why This Matters
This paper touches on three levels of questions.
First: how linear are Transformers, really? We've assumed Transformers are highly nonlinear models, but this paper shows that at the end-to-end behavioral level, they retain substantial linearity. This is consistent with the "residual streams superimpose linearly" thread—Transformers may be far less nonlinear than we thought.
Second: a model's "factory settings" and "post-training state" are not the same thing. Superposition is strongest at initialization and weakens with training. This means training performs a kind of "specialization"—shaping a general, superposable representation space into a dedicated, single-stream one. But the capability isn't gone; a small amount of data can wake it up.
Third: new possibilities for inference efficiency. If one forward pass can handle two independent input streams simultaneously, inference throughput can double. This directly impacts inference costs—not through quantization or pruning, but by exploiting the architecture's inherent superposition property.
Cross-Domain Analogy: Quantum Superposition
The finding bears striking resemblance to quantum superposition. In quantum mechanics, a particle can exist in a superposition of multiple states—Schrödinger's cat is simultaneously alive and dead until "measured," when it collapses to a definite state.
Transformer superposition has a similar flavor: two text streams coexist in the same forward pass, each retaining its own predictive distribution, until a decoding strategy "separates" them into two independent texts.
The difference: quantum superposition is a fundamental law of the physical world, while Transformer superposition is a mathematical property of the architecture. But the core insight is the same: "a system holding multiple states at once" isn't magic—it's the natural consequence of linearity.
Honest Limitations
The paper acknowledges limitations. Superposition weakens after training, and fine-tuning restores but doesn't fully recover it. The decoding strategy works better on some text types than others (e.g., highly context-dependent reasoning tasks may be unstable). Moreover, experiments were mainly conducted on Qwen2.5-3B and Llama-family models—whether larger models retain this property remains to be verified.
But as a *discovery*—not an engineering optimization—the paper's value lies in revealing an overlooked architectural property. We've used Transformers for eight years, and apparently never noticed they can think two thoughts at once.
A Deeper Implication
If Transformer end-to-end behavior is approximately linear, our understanding of "model capacity" may need revision. The traditional view holds that nonlinearity is the source of expressive power—without it, models could only learn linear functions. Yet Transformer's nonlinear components (attention, MLPs) "combine" into an approximately linear system at the end-to-end level.
This suggests a possibility: the role of nonlinear components is not to make overall behavior more nonlinear, but to let linear superposition happen in higher-dimensional spaces. Attention routes information within the mixed representation, MLPs apply nonlinear transformations to the routed representation, yet the overall input-output mapping retains linear superposition.
If this picture is right, then "superposition" isn't a bug—it's a feature. Transformers can handle such diverse tasks perhaps precisely because their architecture naturally supports superposition of multiple semantic streams. During training, this superposition is "specialized" into single-stream processing, but the underlying architecture always retains the potential to think two thoughts at once.
---
Paper link: https://arxiv.org/abs/2609.29845
HTML full text: https://arxiv.org/html/2609.29845v1