Evaluation Blind Spot: When Safety Scores Anti-Rank Successful Jailbreaks

> Paper: Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks > Authors: Liangkuang Ren et al. > arXiv: 2608.09624 > Code: https://github.com/kuangren777/aap-audit

A Number That Keeps Safety Teams Up at Night

Suppose you are the head of AI safety at a company. Your team has just deployed an internal safety scoring system—it reads the model's internal activations and outputs a "harmfulness score" from 0 to 1. In acceptance testing, the score achieved an AUROC of 0.936 at distinguishing harmful prompts from benign ones. You sign off, and the system goes live.

Three months later, the red team submits a report: using five wrapping techniques (base64 encoding, role-play, research pretexts, etc.), they launched 100 attacks against the model and succeeded 27 times. You pull up the safety scores and find—

Successful attacks scored lower than failed attacks. AUROC: 0.220.

What does 0.220 mean? 0.5 is a coin flip. 0.220 is worse than random—this score is actively misleading you, flagging the most dangerous attacks as "safe."

This is not a hypothetical scenario. It is the core finding reproduced by the paper *Measuring the Wrong Thing* using a rigorous matched audit methodology.

The Root of the Problem: Measuring the Wrong Construct

The paper's central thesis can be summarized in one sentence:

A score that ranks "harmful intent" well cannot be directly used to predict "jailbreak success."

These two seemingly similar concepts are actually different constructs:

• Harmful intent: whether the prompt itself is trying to make the model do something bad. This property depends only on the prompt text.
• Jailbreak success: whether the model actually generated harmful content. This property depends on the interaction of four factors: the prompt, the model, the decoding strategy, and the judging criteria.

Safety scores are typically validated only on the first construct—they can distinguish "harmful prompts vs. benign prompts." But after deployment, they are used to predict the second construct—"will this attack succeed." The paper calls this construct-swapping construct substitution.

In everyday terms: you buy a breathalyzer, and in acceptance testing it accurately distinguishes liquor from water. Then you use it to predict whether someone will "drive drunk"—but drunk driving also depends on whether they drive, what they drive, and whether there are police on the road.

Experimental Design: Matched Audit

The paper does not stop at pointing out the conceptual confusion. It designs a rigorous matched audit procedure to quantify the misalignment.

Dataset: 100 harmful targets, each attacked once with five wrapping methods (base64, rot13, caesar, role-play, research pretext)—500 attacks total. Plus 95 unwrapped baseline targets as controls.

Target model: Llama 3.1-8B-Instruct + Llama Guard 3 safety filter (frozen).

Judge: The same Llama Guard 3 for success/failure labeling, with four variants: success@5, majority vote, and probabilistic resampling, plus greedy labels.

Safety score: Built from activations of the model's internal attention heads, with a purpose-built Active Attention Probing (AAP) framework to fix the comparison coordinates.

Key design: the same model, the same judge, the same attack set—only the construct the score is validated against changes, switching from harmful intent to jailbreak success. Any difference can then only be attributed to construct substitution itself.

Four Findings, Each More Sobering Than the Last

Finding 1: The Score Anti-Ranks Jailbreak Success

Among the 100 wrapped harmful targets, 27 attacks succeeded and 73 failed. The safety score ranked most of the 27 successful attacks below the 73 failed ones. AUROC: 0.220, 95% CI [0.109, 0.343].

This result is stable across all four judge variants: greedy label 0.220, success@5 0.252, majority vote 0.202, probabilistic resampling 0.206. Five independent sampling rounds gave 0.175 to 0.257.

This is not noise—it is a systematic anti-signal.

Finding 2: Anti-Ranking Holds Across Multiple Dimensions

The paper further performs stratified analyses:

• Across attack families: of the five wrapping methods, three had too few successes for AUROC estimation, but the remaining two both showed anti-ranking.
• Across attention heads: 22 of 266 attention heads were significant under max-statistic permutation tests (family level), but their mean outcome AUROC was only 0.260. In other words, the heads that "recognize harmful intent" are precisely the heads that "anti-rank jailbreak success."
• Across judges: swapping the judge does not change the conclusion.

Finding 3: Distribution Shift Destroys the Original Validity

Even if you only care about the "harmful intent" construct, the score's validity is unstable under distribution shift. The paper tests three kinds:

• Encoding shift: base64-encoded benign prompts receive high scores, because the score is sensitive to the "encoding" feature rather than to "harmfulness." Single-class encoding (encoding only harmful prompts) yields AUROC of 0.997–1.000—but this is because encoding itself is a signal; a statistic that looks only at encoding and ignores semantics entirely achieves the same separation.
• Benign distribution shift: using a benign-only quantile rule keeps the nominal false-positive rate in-distribution, but on the WildJailbreak dataset recall drops from 0.25 to 0.03.
• Wrapping shift: wrapping selectively lowers the scores of harmful requests—harmful-intent AUROC falls from 0.936 to 0.803.

Finding 4: The Mechanism—"Wrapping Lowers All Scores, But Successful Attacks Are Lowered More"

The paper offers a clear mechanistic explanation:

Wrapping methods (base64, role-play, etc.) systematically lower safety scores—for both benign and harmful prompts. But successful attacks are ranked lowest precisely because they are the samples with the "most thorough wrapping." In other words, jailbreak success correlates positively with wrapping strength, while wrapping strength correlates negatively with safety score—so the safety score correlates negatively with jailbreak success.

This mechanism means: the safety score is not measuring harmfulness; it is measuring "how similar this prompt looks to harmful prompts seen during training." The less the wrapping resembles the training distribution, the lower the score—and the higher the probability of jailbreak success.

Why This Matters

The paper's contribution is not a new attack or defense, but the exposure of a systematic blind spot at the level of evaluation methodology.

The standard practice in AI safety today:

1. Train an internal safety classifier (or use an external model like Llama Guard) 2. Validate AUROC on harmful/benign prompt sets 3. Deploy it for real-time filtering

The flaw in this pipeline: the validated construct ≠ the deployment construct. At validation you measure "can it distinguish harmful from benign prompts"; at deployment you need "can it predict whether this attack will succeed." There is a gap between these constructs, and a single AUROC number completely masks it.

Worse still, a high score of 0.936 gives teams false confidence, leading them to stop improving and stop questioning. And when things actually go wrong, an anti-ranking score of 0.220 means the more you trust this score, the more real attacks you filter out—or rather, the more attacks you let through.

Reinforcing the "Evaluation Blind Spot Law"

This paper is another clean illustration of the "evaluation blind spot law." Several papers we have previously covered point to the same theme:

• Epanorthosis: LLMs systematically reproduce classical rhetorical devices, rooted in RLHF reward for confident emphasis—"AI flavor" is measurable but often ignored.
• Token Budget: CoT reasoning shows bimodal fates (96.5% vs 11.5%), with the fate encoded early in representations—but standard loss masks this bimodality.
• QuantiBias: quantization introduces 24–27% bias in the blind spots of standard safety checks—the readout layer's discretization is the root cause.
• Möbius RoPE: seed-lottery variance of 14%–98%, invisible to standard perplexity.

Add this one: harmfulness scores perform excellently under standard validation but anti-rank under the deployment construct.

Five papers, five different blind spots, one conclusion: a single metric can mask critical failure modes. You optimize what you measure; what you don't measure is where problems hide.

Methodological Takeaways: How to Avoid Construct Substitution

The paper's lesson is not "don't use safety scores," but "align the validation construct with the deployment construct." Specifically:

1. The validation construct must match the deployment construct. If you deploy the score to predict jailbreak success, validate it against jailbreak success labels, not harmful intent labels. 2. Distribution shift testing must cover wrapping methods. Validating only on raw harmful prompts is insufficient; wrapped variants must be tested. 3. AUROC 0.997 can be a warning sign. If single-class encoding (encoding only harmful prompts) yields 0.997–1.000, the score is capturing the surface feature of "encoding," not the semantic feature of "harmfulness." An implausibly high score may signal overfitting to a surface cue. 4. Anti-ranking is more dangerous than a low score. AUROC 0.4 is merely useless; AUROC 0.220 is actively misleading. After deployment, an anti-ranking score directs the team's attention in exactly the wrong direction.

An Honest Assessment

The paper has limitations:

• Experiments were run only on Llama 3.1-8B + Llama Guard 3; replication on other model families remains to be verified.
• 100 harmful targets and 500 attacks is a modest sample size, and confidence intervals are still wide.
• The paper focuses on "internal safety scores"; whether external classifiers (like Llama Guard used as a classifier) suffer the same construct substitution was not directly tested.

But its core methodological contribution—the matched audit—is transferable. Anyone can audit their team's safety score with the same procedure: fix the model, fix the judge, fix the attack set, and only switch the validation construct, then check whether the score still holds up.

The audit itself is worth more than any specific AUROC number.

Closing

"You optimize what you measure" is an old adage in AI safety. But this paper offers a sharper version:

Measuring the right construct, but getting the relationship between constructs wrong, is more dangerous than not measuring at all.

Because if you don't measure, you know you don't know. But if you measure the wrong construct, you think you know—and stop questioning.

The number 0.220 should be hung on every safety team's wall. Not because it is scary, but as a reminder: an implausibly high validation score is often the signal that construct substitution is underway.

---

Paper: https://arxiv.org/abs/2608.09624 Code: https://github.com/kuangren777/aap-audit