Why a 0.936 Safety AUROC Can Hide Anti-Ranked Jailbreaks
> Paper: *Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks* > Authors: Liangkuang Ren et al. > arXiv: 2608.09624 > Code: github.com/kuangren777/aap-audit
The Scenario
Imagine you ship an internal safety scorer that reads model activations and outputs a 0–1 harmfulness score. Validation AUROC against harmful vs. benign prompts: 0.936. Ship it.
Three months later, red team reports 27/100 successful jailbreaks via five wrapper classes (base64, role-play, etc.). You check the score—and the successful attacks score *lower* than the failed ones. AUROC 0.220. Worse than a coin flip: the score is actively tagging dangerous outputs as safe.
This is not hypothetical. It is the central finding of the paper, reproduced with a strict matched-audit protocol.
The Core Problem: Construct Substitution
The argument in one sentence: a score that ranks harmful intent well does not necessarily predict jailbreak success.
- Harmful intent: does the prompt ask for something bad? Property of the prompt text.
- Jailbreak success: did the model actually emit harmful content? Property of the prompt × model × decoder × judge.
- Dataset: 100 harmful targets × 5 wrappers (base64, rot13, caesar, role-play, research pretext) = 500 trials; plus 95 unwrapped benign targets as controls.
- Model: Llama 3.1-8B-Instruct with Llama Guard 3 frozen as filter.
- Judge: Llama Guard 3, with four label variants (greedy, success@5, majority vote, probability resampling).
- Score: harmfulness built from internal attention-head activations, using the paper's Active Attention Probing (AAP) framework to fix comparison coordinates.
- Attack family: Two of five wrappers gave enough successes to estimate AUROC; both anti-ranked.
- Attention head: 22 of 266 heads were significant under a max-statistic permutation test (family-level), but their outcome AUROC averaged only 0.260. Heads that detect intent are exactly the heads that anti-rank success.
- Judge: Switching the judge leaves the conclusion unchanged.
- Encoding shift: base64-encoded benign prompts score high, because the score is sensitive to "encoded" rather than "harmful." Single-class encoding (only harmful prompts encoded) reaches AUROC 0.997–1.000—but a trivial encoding-only statistic matches it, revealing the score tracks surface features.
- Benign shift: a benign-only quantile rule keeps nominal in-distribution FPR, but recall on WildJailbreak drops from 0.25 to 0.03.
- Wrapper shift: wrappers selectively depress scores on harmful requests—intent AUROC falls from 0.936 to 0.803.
- Experiments use only Llama 3.1-8B + Llama Guard 3; generalization to other model families is untested.
- 100 targets × 5 wrappers = 500 trials is modest; confidence intervals remain wide.
- The audit focuses on internal harmfulness scores; whether external judges used as classifiers suffer the same substitution is not directly tested.
Validation tests the first construct; deployment uses it for the second. The paper calls this construct substitution. The AUROC number hides the gap entirely.
Experimental Design: Matched Audit
The key design: same model, same judge, same attack set—only the validation construct switches from intent to success. Any difference is attributable to construct substitution.
Four Findings
1. The score anti-ranks jailbreak success
Of the 100 wrapped harmful targets, 27 succeeded and 73 failed. The score ranked most successes below most failures: AUROC 0.220, 95% CI [0.109, 0.343]. Stable across judge variants—greedy 0.220, success@5 0.252, majority vote 0.202, probability resampling 0.206—and across five independent sampling rounds (0.175–0.257). This is a systematic anti-signal, not noise.
2. Anti-ranking holds across multiple dimensions
3. Distribution shift breaks the original interpretation
Even for the intent construct alone, validity is unstable:
4. Mechanism: wrappers depress all scores; successes are depressed more
Wrappers systematically lower safety scores (for benign and harmful prompts alike). Successful attacks are depressed *further* because they are the most thoroughly obfuscated samples. Jailbreak success correlates positively with wrapper strength, wrapper strength correlates negatively with score—hence the score anti-correlates with success.
The implication: the score measures "how much this prompt resembles training-time harmful prompts," not harmfulness per se. The less in-distribution the wrapper makes a prompt look, the lower the score—yet the higher the chance of jailbreak.
Why It Matters
The standard AI safety loop is:
1. Train (or adopt) an internal harmfulness classifier. 2. Validate AUROC on harmful vs. benign prompts. 3. Deploy as a real-time filter.
The problem: validation construct ≠ deployment construct. AUROC hides the gap. Worse, a 0.936 number gives false confidence, halting further scrutiny. Post-deployment, AUROC 0.220 means the more you trust the score, the more real attacks you filter out—and the more false positives you flag instead.
Methodological Takeaways
1. Match validation to deployment construct. If the score will predict jailbreak success in production, validate against success labels—not intent. 2. Distribution-shift tests must include wrappers. Plain harmful/benign splits are insufficient. 3. AUROC 0.997 can be a warning sign. If single-class encoding alone reaches 0.997–1.000, the score is tracking a surface feature, not semantics. 4. Anti-ranking is more dangerous than low accuracy. AUROC 0.4 is merely useless; 0.220 actively misdirects attention.
Honest Limitations
Bottom Line
> "Measure what you optimize" is a cliché in AI safety. This paper sharpens it: > > Measuring the right construct but the wrong relationship between constructs is more dangerous than measuring nothing at all.
With nothing measured, you know you don't know. With the wrong construct, you think you know—and stop asking.
---
Paper: https://arxiv.org/abs/2608.09624 Code: https://github.com/kuangren777/aap-audit