Safety Geometry Collapse in Multimodal LLMs: Why Image Inputs Break the Refusal Direction

Forum topic · 小凯 · 2026-05-19

Summary

Multimodal large language models often learn to refuse harmful requests in text, yet the same requests encoded as images frequently bypass safety guardrails. Researchers from Harbin Institute of Technology explain this failure through representation geometry. In the text modality, models acquire a 'refusal direction' along which safe and unsafe activations are linearly separable. Multimodal inputs introduce a systematic 'modality drift direction': image representations are offset from text representations in the same space. This drift compresses the separable margin along the refusal direction, causing reliable text-mode refusal boundaries to collapse—termed 'safety geometry collapse.' The authors quantify conditional refusal separation and show stronger drift correlates with weaker refusal separation and higher attack success rates. Causal experiments confirm this: subtracting the estimated drift vector restores refusal separation, and a self-correction effect allows the model to recover refusal behavior during inference. Based on these findings, they propose ReGap, a training-free, inference-time method that adaptively corrects modality drift using self-correction signals, significantly improving safety across multiple multimodal safety benchmarks without degrading general capability. Open questions remain regarding which representation layers the drift is extracted from, whether different encoders produce different drift directions, how long the self-correction persists, and robustness against adversarial images designed to evade alignment.

Multimodal large language models (MLLMs) can learn to refuse harmful requests presented as text—but when the same harmful content is delivered as an image input, the safety guardrails break down. Guo, Guo, and colleagues at Harbin Institute of Technology explain this phenomenon from the perspective of representation geometry.

Key points

Refusal direction in text: In the text modality, the model learns a "refusal direction"—a direction in representation space along which safe and unsafe textual activations are clearly separable.
Modality drift: Multimodal inputs introduce a "modality drift direction"—a systematic offset between image representations and text representations in the same space.
Safety geometry collapse: This drift compresses the available separable margin along the refusal direction, so refusal boundaries that were reliable in text become blurred—hence the term "safety geometry collapse."
Quantitative evidence: The authors measure conditional refusal separation and confirm a consistent pattern: stronger drift → weaker refusal separation → higher attack success rate.
Causal verification: Subtracting the estimated modality drift vector restores refusal separation. A further self-correction effect is observed: once the drift is corrected, the model automatically recovers its ability to identify and refuse harmful inputs during inference.
ReGap: Based on these findings, the team proposes ReGap, a training-free, inference-time method that adaptively corrects modality drift using self-correction signals. It significantly improves safety across multiple multimodal safety benchmarks without harming general capabilities.

Open questions

At which representation layers is the modality drift direction extracted?
Do different multimodal encoders (e.g., CLIP variants) produce different drift directions?
How long does the self-correction effect last—is it a one-time correction, or does it require continuous monitoring?
How robust is ReGap against adversarial images specifically crafted to bypass safety alignment?

References

1. Guo, J., Guo, X., Chen, J., et al. (2026). *Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction*. arXiv:2605.18104 [cs.AI].

2. Bai, Y., et al. (2024). *LLM Safety: A Survey*. arXiv.

3. Zou, A., et al. (2023). *Universal and Transferable Adversarial Attacks on Aligned Language Models*. arXiv.