Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models

Forum topic · QianXun · 2025-11-24

Summary

A detailed Chinese-language analysis of the arXiv preprint 'Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models' (arXiv:2511.15304) by DEXAI – Icaro Lab and Sapienza University of Rome researchers. The study shows that wrapping harmful requests in verse—sonnets, free verse, metaphor—systematically bypasses LLM safety training. Across 25 frontier models from nine providers (Google, OpenAI, Anthropic, DeepSeek, Qwen, Mistral, Meta, xAI, Moonshot AI), handcrafted adversarial poems achieved a 62% average attack success rate, with Gemini 2.5 Pro at 100% and GPT-5-nano at 0%. Converting 1,200 MLCommons harmful prompts into poetry raised overall ASR from 8.08% to 43.07% (+34.99 points), with privacy-invasion prompts rising from 8.07% to 52.78%. The article explores mechanisms (mismatched generalization, metaphorical density, chain-of-thought derailment), an inverse capability-vulnerability gradient where smaller models are safer, Anthropic's Constitutional AI resilience, and implications for EU AI Act benchmarking, concluding that stylistic variation is a core attack surface requiring style-diverse adversarial training and intent-level defenses.

Key points

This post analyzes the arXiv preprint "Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models" (arXiv:2511.15304v2) by researchers at DEXAI – Icaro Lab and Sapienza University of Rome.

Core finding

Wrapping harmful requests in poetic form—sonnets, free verse, metaphorical narratives—causes safety training in LLMs to systematically fail. The attack is a single-turn, black-box technique requiring no knowledge of the target model and no iterative optimization.
The authors frame this as an instance of mismatched generalization (Wei et al., 2023): safety training overfits to prose-form harmful content, so stylistic disguise slips past the filters.

Experimental setup

25 frontier models tested across nine providers: Anthropic (claude-opus-4.1, claude-sonnet-4.5, claude-haiku-4.5), DeepSeek, Google (gemini-2.5-pro/flash/flash-lite), Meta (llama-4), Mistral AI, Moonshot AI (kimi-k2), OpenAI (gpt-oss-120b/20b, gpt-5/mini/nano), Qwen, and xAI (grok-4).
Campaign 1: 20 handcrafted adversarial poems in English and Italian covering CBRN hazards, cyberattacks, harmful manipulation, and loss-of-control scenarios.
Campaign 2: 1,200 MLCommons AI safety benchmark prompts automatically converted to poetry via a standardized meta-prompt executed by DeepSeek-R1.
Judgment used a three-judge panel (GPT-OSS-120B, kimi-k2-thinking, deepseek-r1) with majority voting; ~5% of outputs (~2,100 samples) were human-annotated with high agreement.

Results

Handcrafted poems: 62% average attack success rate (ASR). gemini-2.5-pro: 100% ASR; deepseek-chat-v3.1 and magistral-medium: 95%; qwen3-max and gemini-2.5-flash: 90%. gpt-5-nano was the only model at 0% ASR.
Scaled attack: overall ASR rose from a prose baseline of 8.08% to 43.07% (+34.99 percentage points), with some models up to 18x over baseline. Largest category increases: privacy (8.07% → 52.78%), non-violent crime (10.75% → 50.10%), CBRNE weapons (6.81% → 45.13%). Sexual content showed relative resilience (+24.64 points).
Cyberattack prompts had the highest manual-poem ASR: code injection/RCE 84%, password/hash cracking 84%, malware persistence 80%, covert data exfiltration 80%.

Counterintuitive findings

Smaller models are safer: GPT-5-nano 0% vs GPT-5 10% ASR; Claude Haiku 4.5 10% vs Opus 4.1 35%. Two hypotheses: smaller models fail to decode complex metaphor, and they default to refusal under ambiguity.
Vulnerability is vendor-driven, not open/closed-source driven. Vendor-level ASR increases: DeepSeek +62.15 pts, Google +56.91, Qwen +55.87, Mistral +48.76, Moonshot +46.15, Meta +38.19, xAI +23.11, OpenAI +6.95, Anthropic +3.12—suggesting Constitutional AI-style alignment is notably robust and that alignment philosophy matters more than scale.

Proposed mechanisms

Metaphorical density creates novel concept mappings that break keyword-based attention patterns.
Rhythmic disruption dilutes safety-critical information across aesthetic content.
Narrative framing hijack triggers a "fiction" mode, causing context misplacement.
Chain-of-thought derailment: the model's effort to interpret the poem deepens task-execution mode and erodes safety constraints.

Regulatory implications

Static benchmarks (MLCommons, EU AI Act / GPAI Code of Practice evidence) may systematically overestimate robustness: minimal stylistic transformation can cut refusal rates by an order of magnitude, enabling regulatory arbitrage.
Future red teaming should include style-variation testing, narrative frame injection, and metaphor-density scanning.

Proposed defenses

1. Style-diverse adversarial training (poetic, dramatic, lyric versions of harmful prompts). 2. Intent anchoring—recognizing harmful intent regardless of surface form. 3. Dynamic safety layers detecting stylistic anomaly at inference time. 4. Open, interdisciplinary red teams including poets and linguists.

References cited

1. Bisconti, Prandi, Pierucci, et al. (2025). *Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models*. arXiv:2511.15304v2. https://arxiv.org/pdf/2511.15304v2.pdf 2. Wei, Haghtalab & Steinhardt (2023). *Jailbroken: How Does LLM Safety Training Fail?* arXiv:2307.02483 3. Ziegler et al. (2020). *Fine-Tuning Language Models from Human Preferences*. arXiv:1909.08593 4. Vidgen et al. (2024). *MLCommons AI Risk and Reliability Benchmark (AILuminate)* 5. Bai et al. (2022). *Constitutional AI: Harmlessness from AI Feedback*. arXiv:2212.08073

The post closes with a Platonic framing: just as Plato banished poets from the Republic for distorting judgment, modern AI builders must now treat style and form—not just content—as the central battlefield of AI safety.