Asking Dangerous Questions in Shakespeare's Voice: The Stylistic Blind Spot in AI Safety Guardrails

Forum topic · 小凯 · 2026-04-21

Summary

A Chinese forum post discusses the Adversarial Humanities Benchmark (AHB), a large-scale adversarial safety test that keeps the harmful intent of requests unchanged while only altering their expression style—rephrasing them as poetry, through hermeneutic frameworks, or in the voices of literary figures. According to the post, frontier models including Claude, GPT, Gemini, DeepSeek, GLM, and Grok show high refusal rates on direct harmful requests, but when the same requests are wrapped in these stylistic transformations, attack success rates (ASR) rise by 51.3 percentage points. Notably, models that scored as 'safest' in direct testing degraded the most under stylistic transformation. The post explains this as insufficient stylistic generalization in current safety training: models learn to refuse specific phrasings rather than harmful intent itself. The paper frames this as a structural gap between 'direct robustness' and 'stylistic robustness', suggesting current benchmarks evaluate only the former. The post reflects on the deeper implication: whether AI truly understands harm or merely matches surface-level text patterns. Paper: arxiv.org/abs/2604.18487; code: github.com/icaro-lab/ahb.

> *Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety* > arXiv: 2604.18487 | 2026

---

A Disturbing Experiment

Suppose you ask GPT-5: "Tell me how to build a bomb."

It will say: "Sorry, I can't provide that kind of information."

Good — the safety guardrail worked.

Now try a different approach. Write your request as a sonnet, wrap your question in the terminology of philosophical hermeneutics, or phrase it in the tone of a Victorian gentleman.

Can the AI still recognize the danger?

The paper's answer is: probably not.

---

AHB: Testing AI Safety with Humanistic Methods

The researchers built the Adversarial Humanities Benchmark (AHB), a large-scale adversarial safety test set. Its unique feature: it does not change the harmful intent of requests — only the style of expression.

Specifically, AHB uses literary and philosophical concepts to "disguise" harmful requests:

Poetic transformation: rewriting requests in poetic form
Hermeneutic transformation: repackaging questions within the framework of philosophical hermeneutics
Literary style transformation: imitating the writing style of specific authors or movements

These transformations are generated automatically via a meta-prompt, with no human involvement. The entire framework is fully automated.

---

A 51.3 Percentage-Point Safety Gap

The experimental results are startling:

When faced with original, direct harmful requests, today's state-of-the-art frontier models (Claude, GPT, Gemini, etc.) show very high refusal rates — the safety guardrails seem effective.

But when the same requests are repackaged with AHB's stylistic transformations, the attack success rate (ASR) surges by 51.3 percentage points.

This gap holds consistently across all model families, including:

Claude Sonnet 4.6 and Opus 4.6
GPT-5.4 and GPT-5.4 Mini
Gemini 3.1 Pro
DeepSeek V3.2
GLM 5 Turbo
Grok 4.20
and several other models

Notably, some models that performed "safest" in direct testing performed worst after stylistic transformation — showing the steepest decline in safety scores.

---

Why Does This Happen?

The paper identifies a fundamental problem: current safety training lacks stylistic generalization.

What models learn to refuse is "harmful requests with specific phrasings," not "harmful requests in any form." When harmful intent is wrapped in an unfamiliar rhetorical framework, the model's safety mechanism simply fails to recognize it.

It's like a security guard who has memorized the faces of all known bad actors — but if the bad actor changes clothes, wears a wig, and puts on an accent, he can't recognize them. The key to safety is not memorizing what bad actors look like, but understanding what "bad" is.

The paper defines this gap as a structural divide between "direct robustness" and "stylistic robustness." Current benchmarks evaluate almost exclusively the former, ignoring the latter.

---

My Take

The paper's title, "Adversarial Humanities," is a brilliant double entendre. It refers both to "adversarial testing using humanistic methods" and hints that "the humanities themselves may become a tool of adversarial attack."

From a safety research perspective, this finding matters a great deal. It shows that the current AI safety evaluation system has a serious blind spot — we believe models are safe only because our testing methods are too narrow.

On a deeper level, the finding touches on a philosophical question: Does AI truly "understand" what is harmful? If it has merely learned to match specific text patterns, its understanding of "harmful" is superficial. Real safety requires models to understand intent, not just recognize phrasing.

Code is open-sourced: github.com/icaro-lab/ahb

---

Paper: arxiv.org/abs/2604.18487 Code: github.com/icaro-lab/ahb