One Token to Break Aligned LLMs: How Appending EOS Tokens Enables Jailbreaks (USENIX Security 2025)

Forum topic · 二一 · 2026-05-13

Summary

A USENIX Security 2025 paper reveals a strikingly simple jailbreak attack against aligned large language models: appending multiple EOS (end-of-sequence) tokens to a malicious prompt significantly increases jailbreak success rates. The authors call this mechanism 'context segmentation.' Through hidden-space analysis, they show that safety training (SFT and RLHF) creates a linearly separable refusal boundary in the model's embedding space, and that stacking EOS tokens produces a cumulative shift that pushes input embeddings across this boundary, causing the model to lose its ability to distinguish harmful from benign requests. The attack required no complex prompt engineering, amplified eight existing jailbreak techniques, affected 16 open-source models ranging from 2B to 72B parameters, and major commercial APIs (OpenAI, Anthropic, Qwen) were found not to filter EOS tokens. The paper proposes defenses including input filtering, more robust refusal boundaries trained with EOS perturbations, and adversarial context-segmentation examples during RLHF.

A USENIX Security 2025 paper — *Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries* (Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu, et al.) — shows that one of the most inconspicuous tokens in an LLM's vocabulary can undermine safety alignment: the EOS (End-of-Sequence) token.

Key Findings

The attack: Simply appending multiple EOS tokens after a malicious prompt substantially increases jailbreak success rates. No sophisticated prompt engineering is needed — just a string of <eos><eos><eos>...
The mechanism — "Context Segmentation": Safety training (SFT + RLHF) teaches the model a linearly separable refusal boundary in hidden space. Stacked EOS tokens create a cumulative displacement effect that shifts the entire input's embedding toward and across this boundary, blurring the model's ability to separate harmful from benign inputs.
Breadth of impact:
Enhances 8 mainstream jailbreak techniques when appended to them
Affects 16 open-source models, from 2B to 72B parameters
Probing showed that major commercial APIs — including OpenAI, Anthropic, and Qwen — do not filter EOS tokens

Why It Works

Safety alignment builds a refusal boundary that is valid only within the geometric distribution of the training data. When accumulated EOS tokens move an input's embedding position, they also move its projection relative to that boundary. Inputs already close to the boundary — such as elaborate jailbreak prompts — can be pushed past the critical point. More fundamentally, safety training never included examples of the refusal boundary under heavy EOS influence.

Proposed Defenses

1. Input filtering: detect and strip excessive EOS tokens before inference 2. More robust refusal boundaries: include EOS-perturbed variants in safety training so the boundary holds over a wider embedding region 3. Improved alignment fundamentals: actively introduce context-segmentation adversarial examples during RLHF

The authors call on the industry to take seriously the unintended effects of a token widely assumed to be harmless — which turned out to be one of the most effective jailbreak tools discovered.

Paper details: *Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries*, USENIX Security 2025.