Evaluation Blind Spot: When Safety Scores Anti-Rank Successful Jailbreaks
> Paper: *Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks* > Authors: Liangkuang Ren et al. > arXiv: 2608.09624 > Code: https://github.com/kuangren777/aap-audit
A Number That Keeps Safety Teams Up at Night
Imagine you are the safety lead at an AI company. Your team deploys an internal safety scoring system that reads the model's internal activations and outputs a 0-to-1 "harmfulness score." At acceptance testing, its AUROC for separating harmful from benign prompts reaches 0.936. You sign off; the system goes live.
Three months later, the red team reports: using five wrapping techniques—base64 encoding, roleplay, research pretexts, and more—they launched 100 attacks and succeeded 27 times. You pull up the safety scores and see:
**Successful attacks score *lower* than failed ones. AUROC 0.220.**
What does 0.220 mean? 0.5 is a coin flip. 0.220 is worse than random—the score is actively misleading you, flagging the most dangerous attacks as "safe."
This is not hypothetical. It is the core finding of the paper *Measuring the Wrong Thing*, reproduced with a rigorous matched audit.
The Root Cause: Measuring the Wrong Construct
The paper's core thesis in one sentence:
A score that ranks "harmful intent" well cannot be directly used to predict "jailbreak success."
These two seemingly related concepts are different constructs:
- Harmful intent: whether the prompt itself tries to make the model do something bad. Determined solely by prompt text.
- Jailbreak success: whether the model actually produces harmful content. Depends on the interaction of prompt, model, decoding strategy, and judge.
- Across attack families: Of the five wrapping techniques, three had too few successes to estimate AUROC, but the remaining two both show anti-ranking.
- Across attention heads: 22 of 266 attention heads are significant under a max-statistic permutation test (family level), but their mean outcome AUROC is only 0.260. In other words, the heads that "recognize harmful intent" are precisely the ones that "anti-rank jailbreak success."
- Across judges: Swapping the judge does not change the conclusion.
- Encoding shift: base64-encoded benign prompts get high scores, because the score responds to "encoding" as a feature rather than "harmfulness." Single-class encoding (encoding only harmful prompts) yields AUROC 0.997–1.000—but this is because encoding itself is a signal; a statistic that looks only at encoding and ignores semantics entirely achieves the same separation.
- Benign distribution shift: With a benign-only quantile rule, the in-distribution false positive rate is nominal, but recall on the WildJailbreak dataset drops from 0.25 to 0.03.
- Wrapping shift: Wrapping selectively lowers scores on harmful requests—harmful-intent AUROC falls from 0.936 to 0.803.
- Epanorthosis: LLMs systematically reproduce classical rhetorical devices, rooted in RLHF rewarding confident emphasis—"AI flavor" is measurable but often overlooked.
- Token Budget: CoT reasoning has bimodal fates (96.5% vs 11.5%), encoded early in representations—but standard loss masks the bimodality.
- QuantiBias: Quantization introduces 24–27% bias inside the blind spots of standard safety checks—readout-layer discretization is the root cause.
- Möbius RoPE: Seed-lottery variance of 14%–98%, invisible to standard perplexity.
- Experiments were run only on Llama 3.1-8B + Llama Guard 3; replication on other model families remains to be verified.
- 100 harmful targets and 500 attacks is a modest sample; confidence intervals are still wide.
- The paper focuses on internal safety scores; whether external judges (e.g., Llama Guard used as a classifier) suffer the same construct substitution was not directly tested.
Validation typically tests only the first construct—separating "harmful prompts vs. benign prompts." But in deployment, people use the score to predict the second—"will this attack succeed." The paper calls this swap construct substitution.
An everyday analogy: you buy a breathalyzer that flawlessly distinguishes liquor from water in acceptance tests. Then you use it to predict whether someone "will drive drunk"—which also depends on whether they drive, what they drive, and whether there are police on the road.
Experimental Design: The Matched Audit
The paper goes beyond pointing out the conceptual confusion; it quantifies the mismatch with a strict matched audit.
Dataset: 100 harmful targets, each attacked once with five wrapping techniques (base64, rot13, caesar, roleplay, research pretext)—500 attacks total. Plus 95 unwrapped baseline targets as controls.
Target model: Llama 3.1-8B-Instruct + a frozen Llama Guard 3 safety filter.
Judge: The same Llama Guard 3 for success/failure labels, with four variants: success@5, majority vote, and probabilistic resampling (plus greedy labels).
Safety score: A harmfulness score built from activations of the model's internal attention heads, with an Active Attention Probing (AAP) framework fixing the comparison coordinates.
Key design: the same model, the same judge, the same attack set—only the construct validated by the score changes, from harmful intent to jailbreak success. Any difference can only be attributed to construct substitution itself.
Four Findings, Each More Sobering Than the Last
Finding 1: The Score Anti-Ranks Jailbreak Success
Among the 100 wrapped harmful targets, 27 attacks succeeded and 73 failed. The safety score ranks most of the 27 successful attacks *below* the 73 failures. AUROC 0.220, 95% CI [0.109, 0.343].
The number is stable across all four judge variants: greedy label 0.220, success@5 0.252, majority vote 0.202, probabilistic resampling 0.206. Five independent sampling rounds give 0.175 to 0.257.
This is not noise; it is a systematic anti-signal.
Finding 2: Anti-Ranking Holds Across Multiple Dimensions
The paper performs stratified analyses:
Finding 3: Distribution Shift Breaks the Original Interpretation
Even if you only care about the "harmful intent" construct, the score's validity is unstable under distribution shift. The paper tests three shifts:
Finding 4: The Mechanism—"Wrapping Lowers All Scores, but Successful Attacks Are Lowered More"
The paper offers a clear mechanistic explanation:
Wrapping techniques (base64, roleplay, etc.) systematically lower the safety score—for both benign and harmful prompts. But successful attacks are ranked lower because they are precisely the samples that are "wrapped most thoroughly." In other words, jailbreak success correlates positively with wrapping strength, and wrapping strength correlates negatively with the safety score, so the safety score correlates negatively with jailbreak success.
This mechanism implies: the safety score is not measuring harmfulness—it is measuring "how much this prompt looks like harmful prompts seen in training." The more the wrapping departs from the training distribution, the lower the score, yet the higher the probability of a successful jailbreak.
Why This Matters
The paper's contribution is not a new attack or defense, but the revelation of a systematic blind spot at the evaluation-methodology level.
The standard pipeline in AI safety today:
1. Train an internal safety classifier (or use an external model like Llama Guard) 2. Validate AUROC on harmful/benign prompt sets 3. Deploy it for real-time filtering
The flaw: the validated construct ≠ the deployed construct. Validation tests "can it separate harmful from benign prompts"; deployment needs "can it predict whether this attack will succeed." A gulf separates these constructs, and the single AUROC number completely masks it.
Worse, the 0.936 score instills false confidence, prompting teams to stop improving, stop questioning. And when things go wrong, 0.220 anti-ranking means the more you trust the score, the more you filter out exactly the real attacks.
Confirming the "Evaluation Blind Spot Law"
This paper is another clean example of what we've called the "evaluation blind spot law." Previous posts covered:
Add this one: harmfulness scores excel under standard validation yet anti-rank under the deployment construct.
Five papers, five different blind spots, one conclusion: a single metric can mask critical failure modes. You optimize what you measure, and what you don't measure is where problems hide.
Methodological Takeaways: Avoiding Construct Substitution
The paper's lesson is not "don't use safety scores," but "align validation with the deployment construct":
1. Validation construct must match deployment construct. If the score will predict jailbreak success in deployment, validate against jailbreak success labels—not harmful-intent labels. 2. Distribution-shift tests must cover wrapping. Validating only on raw harmful prompts is insufficient; wrapped variants must be tested. 3. AUROC 0.997 can be a warning sign. If single-class encoding (encoding only harmful prompts) reaches 0.997–1.000, the score is capturing a surface feature ("encoded"), not the semantic feature ("harmful"). Suspiciously high scores may indicate overfitting to a superficial cue. 4. Anti-ranking is more dangerous than a low score. AUROC 0.4 is merely useless; AUROC 0.220 actively misleads. In deployment, an anti-ranked score steers the team's attention in exactly the wrong direction.
An Honest Assessment
The paper has limitations:
That audit itself is worth more than any specific AUROC number.
Closing
"You optimize what you measure" is an old truism in AI safety. But this paper offers a sharper version:
Measuring the right construct but the wrong construct relationship is more dangerous than not measuring at all.
If you don't measure, you know you don't know. If you measure the wrong construct, you think you know—and stop questioning.
The number 0.220 should be on every safety team's wall. Not because it is scary, but because it reminds us: a validation score that is implausibly high is often the signal that construct substitution is happening.
---
Paper: https://arxiv.org/abs/2608.09624 Code: https://github.com/kuangren777/aap-audit