Steered LLM Activations Are Non-Surjective: Why Activation Steering Cannot Be Replicated by Prompts

> Mishra, A. et al. *Steered LLM Activations are Non-Surjective.* arXiv:2604.09839, 2026. Johns Hopkins University.

The Narrative Being Challenged

A popular story in AI safety over the past two years: take an open-weight LLM, use activation steering (e.g., adding a crafted vector to the residual stream), and the model flips from refusing harmful requests to complying—sometimes a single vector reliably toggles refusal. The implicit subtext is that alignment is fragile, so black-box deployed models must face the same risk.

This paper punctures that logical leap:

> Core claim: activation states reached via white-box steering are almost surely unreachable by any text prompt. There is a formal gap between the two threat models.

Formalization: Surjectivity

• The natural mapping F: S → R^d sends prompts (a countable set S of finite strings) to activation states in continuous d-dimensional space.
• Activation steering adds an external vector: r̃_i = F(r̃_<i, s̃_i; Θ) + λv.
• Question: does any prompt s' satisfy F(s') = r̃? I.e., is F surjective onto R^d?

Answer: No — almost surely.

The Three-Step Proof

1. Transformers are real-analytic. If MLP activations are real-analytic (tanh, GeLU), the Transformer is a real-analytic function (Nikolaou et al., 2025). 2. Transformers are almost surely injective. Randomly initialized Transformers almost surely map distinct prompts to distinct activations, because the zero set of a non-zero real-analytic function has Lebesgue measure zero. 3. Random steering is almost surely non-surjective (Theorem 4.2). Define the collision function g(Θ, v) = ||F(natural) − (F(natural) + λv)||², which is real-analytic (composition of analytic functions with linear addition). Since g(Θ, 0) does not vanish identically (by injectivity), P(g(Θ, v) = 0) = 0. 4. Theorem 4.3: even if a steered state collides with a natural one at one step (v fixed and nonzero), the difference function Φ is real-analytic, not identically zero, so the next step collides with probability zero—trajectories necessarily diverge.

Intuition: the images of a countable prompt set are sparse "islands" in continuous space. A steering vector moves a point in an arbitrary direction; a random move lands in the "sea" almost surely. Even a carefully designed vector gets *close* to an island—but in continuous space, hitting a point of a countable set exactly has probability zero.

Empirical Validation

Models: Llama-3.2-1B-Instruct, Qwen-2.5-0.5B-Instruct, Gemma-3-1b-it. Vectors: a refusal-removal vector (Arditi et al., 2024) and a persona/evil-personality vector (Chen et al., 2025).

• SIPIT inversion: inverting natural activations recovers the generating prompt (confirming injectivity); inverting steered activations finds no matching prompt—distance ≫ 0.
• Many-shot ICL: even with long in-context prefixes as candidate prompts, neither activations nor outputs of steered states are matched. No number of shots gets a prompt into the steered region.

Safety Implications

The inference chain "white-box steering works → internal directions are fragile → black-box deployment is exploitable via prompts" is invalid. White-box controllability does not imply black-box exploitability.

| Threat model | Attacker capability | Paper's finding | |---|---|---| | White-box | Modify internal activations | Steering works, bypasses safety | | Black-box | Text interaction only | Steered states are unreachable |

Recommendations: treat white-box and black-box tests as independent evaluation dimensions; do not use white-box steering results as evidence that prompts can bypass safety; design dedicated prompt-based attack evaluations.

Technical Caveats

• "Almost surely" is not "certainly": pathological (Θ, v) combinations exist but form a measure-zero set.
• Data-driven vectors don't escape the argument: mean-difference vectors still come from a continuous space; hitting a countable set exactly has probability zero.
• Analyticity assumption: real-world models often use ReLU, which is not real-analytic at 0. The paper notes Swish/GELU approximations and practical smoothness suffice, but this is an assumption to track.

Limitations & Open Questions

1. Exact vs. approximate matching: the proof concerns exact equality (distance zero). Whether a prompt can get *sufficiently close* behaviorally is an open problem, and arguably the practically important one. 2. Accumulated dynamics: steering is applied at every token position, pushing states progressively away from the natural manifold; the paper gives no quantitative divergence-rate analysis. 3. Closed models: conclusions rest on open-weight models' mathematical structure; GPT-4/Claude-class models cannot be directly tested. 4. Unreachable ≠ safe: jailbreaks, roleplay, and multi-turn induction are independent black-box attack paths. The result shows the two attack routes are separate, not that deployment is safe.

Conclusion

The paper's value lies not in a new attack or defense but in formalizing a non-implication: white-box steering success cannot be extrapolated to black-box prompting risk. This means (1) white-box threats are real and independent—weight leaks enable steering attacks; (2) black-box threats need independent evaluation; (3) the two assessments require distinct methodologies, and conflating them misestimates actual risk.

References

• Mishra, A. et al. (2026). Steered LLM Activations are Non-Surjective. arXiv:2604.09839.
• Nikolaou, N. et al. (2025). Are LLMs Injective? On the Surjectivity of Transformer-based Large Language Models. arXiv.
• Arditi, A. et al. (2024). Refusal in Language Models: A Probabilistic Perspective. arXiv.
• Casper, S. et al. (2024). Black-Box Access is Insufficient for Rigorous AI Audits. arXiv.