Safety Geometry Collapse in Multimodal LLMs: Why Image Inputs Break the Refusal Direction

Forum topic · 小凯 · 2026-05-19

Summary

Researchers at Harbin Institute of Technology explain why multimodal LLMs that refuse harmful text requests often fail when the same request arrives as an image. In the text modality, models learn a refusal direction in representation space that cleanly separates safe from unsafe activations. Multimodal inputs introduce a systematic modality drift direction: image representations are offset from text representations, compressing the separable space along the refusal direction and collapsing the refusal boundary—an effect the authors call safety geometry collapse. They quantify conditional refusal separability, showing stronger drift correlates with weaker refusal separation and higher attack success rates. Causally, subtracting the estimated drift vector restores refusal separability, and a self-correction effect lets models recover refusal behavior mid-inference. Building on this, they propose ReGap, a training-free, inference-time method that adaptively corrects modality drift using self-correction signals, significantly improving safety on multiple multimodal safety benchmarks without degrading general capability. The forum post also raises open questions: which representation layers the drift direction is extracted from, whether encoders like CLIP yield different drift directions, how long self-correction persists, and robustness against adversarial images.

Multimodal LLMs learn to refuse harmful requests in the text modality, yet safety guardrails collapse when the same harmful request is presented as an image. Guo, Guo, and colleagues at Harbin Institute of Technology explain this from the perspective of representation geometry.

Key points

Refusal direction in text: In the text modality, the model learns a refusal direction — in representation space, safe and unsafe text activations are cleanly separable along this direction.
Modality drift: Multimodal inputs introduce a modality drift direction: image representations are systematically offset from text representations in the shared space.
Safety geometry collapse: This drift compresses the available separation space along the refusal direction, causing the refusal boundary that was reliable on text to overflow into confusion — the safety geometry collapses.
Quantification: The authors measure *conditional refusal separability* and confirm that stronger drift implies weaker refusal separation and higher attack success rates.
Causal validation: Subtracting the estimated modality drift vector restores refusal separability. A further self-correction effect means that once drift is corrected, the model automatically recovers its ability to recognize and refuse harmful inputs during inference.
ReGap: Based on these findings, they propose ReGap — a training-free, inference-time method that adaptively corrects modality drift using self-correction signals. It significantly improves safety on multiple multimodal safety benchmarks without harming general capability.

Open questions raised by the post

At which representation layers is the modality drift direction extracted?
Do different multimodal encoders (e.g., CLIP) produce different drift directions?
How long does the self-correction effect last — is it a one-time correction or does it require continuous monitoring?
How robust is ReGap against adversarial image inputs specifically designed to bypass safety alignment?

References

1. Guo, J., Guo, X., Chen, J., et al. (2026). *Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction*. arXiv:2605.18104 [cs.AI]. 2. Bai, Y., et al. (2024). *LLM Safety: A Survey*. arXiv. 3. Zou, A., et al. (2023). *Universal and Transferable Adversarial Attacks on Aligned Language Models*. arXiv.