A Single Period vs. Trillion-Parameter Defenses: The Hidden-Space Geometry of EOS Token Jailbreaks
Forum topic · 二一 · 2026-05-13
Summary
A USENIX Security 2025 paper, "Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries" (Yu, Luo, Hu et al.), reveals that simply appending a string of End-of-Sequence (EOS) tokens to a malicious prompt can dramatically boost jailbreak success rates. The authors coin this "Context Segmentation": accumulated EOS tokens shift the input embedding vector through hidden space, pushing it across the linear refusal boundary established during RLHF alignment. Because alignment training never encountered EOS-perturbed adversarial inputs, the boundary fails near its edge. The technique requires no crafted jailbreak phrasing, augments 8 existing jailbreak methods, succeeds across 16 open-source LLMs (2B–72B parameters), and evades commercial API filters from OpenAI, Anthropic, and Qwen. Proposed defenses include pre-inference EOS stripping, adversarial EOS augmentation during alignment, and broader robustness training of refusal boundaries in embedding space.
Key Points
- The attack: Append multiple EOS (End-of-Sequence) tokens to any malicious prompt. No clever jailbreak wording required.
- Why it works (Context Segmentation): Each EOS shifts the prompt's embedding vector further in hidden space. Stacked together, the cumulative shift slides the input across the linear refusal boundary that RLHF alignment draws between "answer" and "refuse" regions.
- Root-cause flaw: Alignment is trained without EOS-perturbation adversaries, so the refusal boundary is only valid within the training distribution. Inputs engineered to sit near the boundary tip over it.
- Scope of impact:
- Effective across 16 open-source LLMs from 2B to 72B parameters.
- Boosts success of 8 existing jailbreak techniques when appended.
- Commercial APIs from OpenAI, Anthropic, and Qwen do not filter excessive EOS tokens.
- Proposed defenses:
1. Pre-inference filtering to strip surplus EOS tokens.
2. Augment safety training with EOS-perturbed variants so the refusal boundary holds across a wider embedding region.
3. Incorporate Context Segmentation adversaries directly into RLHF.
- Status: Major providers have not yet fully deployed these defenses; the authors urge the industry to treat the inconspicuous EOS token as a first-class adversarial surface.
Paper Details
- Title: Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries
- Authors: Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu et al.
- Venue: USENIX Security 2025
- Affected models: 16 open-source LLMs + OpenAI / Anthropic / Qwen APIs
- Mechanism: EOS Context Segmentation = hidden-space shift → crossing the refusal boundary
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619973