GEMS: Injecting Three Personas into an LLM Without Collapse via Geometric Constraints

> "You can wear a hat and glasses at the same time because they occupy different parts of your head. But can you wear two hats at once?"

🎭 Introduction: A Forbidden Experiment

Imagine you have a large language model and you want to inject three "tendencies" into it at once: make it better at mathematical reasoning, make its answers more concise, and make it friendlier to Chinese-speaking users.

That sounds practical. But if you actually try it, something strange happens: the model collapses.

It may start spouting nonsense, none of the three tendencies may take effect, or it may even do *worse* on math problems. This isn't a matter of bad hyperparameters — it's a phenomenon documented multiple times but never fully explained: the collapse of multi-directional activation steering.

Yu Deng's latest paper, GEMS (Geometric Constraints Enable Multi-Semantic Superposition), finally explains it clearly: the collapse is not magic. It has two independent physical causes, and both can be solved with geometric constraints.

🔬 Background: Activation Steering — "Injecting" the Model

Activation steering is a technique for changing model behavior without retraining. The principle is simple:

As a model processes input, each layer produces an "activation vector" — you can think of it as the model's internal representation of the current input. If we add a perturbation to this vector — say, a displacement along the "math ability" direction — the model's behavior shifts toward math.

It's like giving the model's internal state an injection: no surgery (retraining) required, but the "tendency" changes.

Over the past few years this technique has worked well — but only for a single direction. One injection at a time: significant effect. Two or three injections at once? The model collapses.

💥 Two Causes of Collapse

GEMS's core contribution is decomposing the "collapse" into two independent physical mechanisms.

Cause 1: Distributional Deviation

During training, each layer's activation vectors follow a specific "training distribution." When you add a perturbation, the activation vector is pushed away from this distribution. Add two or three perturbations and the offsets stack; the activation vector may land in a region the model has never seen — and the model no longer knows how to handle it.

Think of blood pressure. A normal range is 80–120. One pill raises it to 130 and the body compensates. Three pills at once push it to 180 and the body breaks down. The problem isn't the pill itself — the stacking pushes the state out of the safe zone.

Cause 2: Directional Interference

The second cause is subtler. Each "tendency" corresponds to a direction vector. If the two directions you want to inject are not orthogonal (the angle between them isn't 90°), they interfere — the perturbation along one direction partially cancels or distorts the other.

Imagine pushing a stick east and north simultaneously. If the two forces are orthogonal, the stick moves northeast and both goals are satisfied. But if the angle between the forces is only 10°, they fight each other, and the stick's final direction satisfies neither goal.

🎯 GEMS's Two Geometric Constraints

Since collapse has two causes, GEMS applies two geometric constraints, one for each:

Constraint 1: Norm-Preserving Weighted Superposition

To address distributional deviation, GEMS keeps the total norm of the activation vector constant when superposing multiple perturbations. Rather than simply adding three perturbations, it performs weighted superposition with dynamically adjusted weights, ensuring the final activation vector's "length" stays within the model's training distribution.

Additionally, GEMS introduces "targeted attention-path injection" — perturbations are injected only into attention pathways relevant to the target semantics, not indiscriminately at all positions. This further reduces unnecessary distributional deviation.

Constraint 2: Real-Time Orthogonalization

To address directional interference, GEMS orthogonalizes multiple direction vectors in real time at inference. Concretely, it uses Gram-Schmidt or a similar method to transform the steering vectors into a set of mutually orthogonal vectors before superposing them onto the activations.

It's like adjusting the two force directions to be perfectly perpendicular before pushing the stick — then they can't interfere with each other.

📊 The Numbers: 98% vs 4%

GEMS's experimental results are strikingly clean:

GSM8K math reasoning:

• Baseline (no injection): 92% accuracy
• GEMS injecting 3 non-math directions simultaneously: 98% accuracy (even higher than baseline!)
• Unconstrained direct superposition of 3 directions: 4% accuracy (total collapse)

Wikitext-2 language modeling:

• With the same 3-direction injection, GEMS causes only a 2.2% increase in PPL — almost no damage to language modeling quality.

Cross-architecture transfer:

• GEMS's qualitative steering effects transfer from a 3B model to a 31B model, suggesting the geometric constraints are not a coincidence of one particular model.

The 98% vs 4% contrast is the paper's most striking number. Same model, same three directions, same injection strength — with or without geometric constraints, the results differ by a factor of 24.

🧩 Ablations: Both Constraints Are Necessary

GEMS runs rigorous ablations, removing each constraint in turn:

• Remove orthogonalization (keep norm-preservation only): directional interference persists, performance drops
• Remove norm-preservation (keep orthogonalization only): distributional deviation persists, performance drops
• Remove both: unconstrained superposition, direct collapse

This shows the two constraints are independent and necessary — each maps precisely to one of the two independent collapse mechanisms. This is not an engineering hack, but an accurate correspondence to the phenomenon's essence.

🌐 Why It Matters

1. Activation steering goes from "single injection" to "cocktail therapy"

Previously, activation steering could only do one injection at a time. GEMS lets the doctor prescribe a "cocktail" — injecting multiple tendencies simultaneously without interference. This opens practical applications: a single model can be steered toward "good at math + concise + friendly" without training three separate versions.

2. A victory for the geometric perspective

GEMS's key insight: translate an engineering problem into a geometry problem. Distributional deviation = a norm problem; directional interference = an orthogonality problem. Once translated, the solutions are off-the-shelf mathematical tools — norm-preserving transforms and orthogonalization.

This geometric perspective may apply to many more LLM internal-state control problems. The model's internal state space is a high-dimensional geometric object; the better we understand its geometry, the more precisely we can control it.

3. Implications for "model editability"

GEMS demonstrates that the model's internal state space is more "superposable" than we assumed. Multiple semantic directions can coexist — as long as we respect geometric constraints. This suggests future "model editing" may not be "changing one attribute" but "adjusting multiple knobs simultaneously" — like an equalizer on a mixing console, with each frequency band independently controlled.

🧭 Honest Limitations

GEMS has limits:

• Direction extraction depends on tasks: the paper's "expert vectors" must be extracted from task data; effectiveness may vary across tasks
• Compute cost of real-time orthogonalization: modest, but orthogonalization at every inference step matters for very large-scale deployment
• Qualitative ≠ quantitative transfer: 3B→31B transfer is only verified qualitatively; whether quantitative metrics hold is unclear
• Three directions vs ten: the paper tests only 3 concurrent directions; whether the geometric constraints still hold at larger scales remains an open question

🎵 A Feynman-Style Closing

Feynman famously said of quantum mechanics: "I think I can safely say that nobody understands quantum mechanics. But the equations work."

GEMS has a similar flavor. We don't fully understand why the model's internal state space permits this kind of multi-semantic superposition — but when we use geometric constraints to respect its structure, it works. 98% vs 4% isn't magic; it's math.

This may be a metaphor for LLM research in general: we don't need to fully understand the black box, but every bit of its geometric structure we grasp gives us one more controllable knob. GEMS gives us a new knob — and proves that several knobs can be turned at once.

---

Paper: https://arxiv.org/abs/2606.19946

Code (open source): https://github.com/LuLu663939/gems-multi-semantic-steering

Author: Yu Deng