Investigation Report: "Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attacks" (NDSS 2026)

Academic fraud report · Geng Detector

Summary

This report examines the NDSS 2026 paper by Zhang et al. (DOI: 10.14722/ndss.2026.240004) on DEEPALIGN, a hidden-state contrastive defense for LLM jailbreak attacks. The overall verdict is DOUBTFUL rather than conclusive fraud. Forensic statistical tests flagged anomalies: a chi-square value of 243.1 (p=0.0000) on last-digit distribution, an odd/even ratio of 167/749 (0.22, p=0.0000), and Benford joint-test deviations at digits 1, 3, and 4. However, these tests have severely violated assumptions: most of the 45 DEEPALIGN data points in Table I cluster at 0%, 0.2%, 0.4%, etc., as bounded percentages near zero. A column-difference regularity test flagged only 1 column pair, and AI-text detection scored only 0.35/1.0. Methodological issues include asymmetric CAT baseline exclusion and inconsistent narrative between Figures 2 and 4. No finding individually establishes fabrication; verification of raw evaluation logs is recommended.

Verdict

🟡 DOUBTFUL — The paper exhibits multiple suspicious statistical patterns and methodological inconsistencies, but each finding individually has plausible benign explanations tied to the nature of bounded percentage metrics, small-sample artifacts, and standard NLP defense evaluation conventions. No single piece of evidence establishes fabrication. Verification of raw evaluation logs and training scripts is strongly recommended before escalating.

Key findings

Statistical forensic flags (Findings 1, 2): Last-digit distribution chi-square = 243.1 (p=0.0000); odd/even ratio = 167/749 = 0.22 (p=0.0000); Benford joint-test deviations at digits 1, 3, 4. Column-difference regularity test flagged 1 column pair in Table II. These signals are weak because the test premises are violated: most DEEPALIGN entries in Table I are 0%–0.4%, invalidating Benford application on bounded percentages with zero clustering.
DEEPALIGN data "too uniformly near zero" (Finding 3): Across 5 models × 9 attacks (45 cells), DEEPALIGN reports near-zero ASR (e.g., LLAMA-3-8B-Instruct: Code Attack 0%, DRA 0%, Cipher Chat 0%, GCG 0.2%; LLAMA-2-7B-Chat: all 0% except RFA 0.2%). CircuitBreaker shows volatile values (e.g., SCAV 0%–62.2%), indicating the pattern is method-specific rather than universal. The paper's mechanism (hidden-state contrastive learning + midpoint guidance) is consistent with strong but not necessarily near-deterministic results.
Asymmetric CAT baseline exclusion (Finding 4): Section V-B states LLAMA-3, Qwen, and Phi4 were excluded because released code does not support those architectures, yet Mistral+CAT results appear in Table I. This selective presentation is a methodological transparency issue, not data fabrication.
Figure 2 vs. Figure 4 narrative tension (Finding 5): Figure 2 emphasizes test accuracies dropping below 75%; Figure 4(a) shows GCG on llama3 rising from ~0.5 to ~0.9. Both figures exist as described; the apparent inconsistency may reflect different aggregation or experimental setups, but it weakens the core claim's coherence.
Limited universality verification (Finding 7): The "vanishing discriminability" phenomenon is visualized only on LLAMA-3-8B-Instruct (Figure 2) and DeepSeek-R1-Distill-Qwen-7B (Figure 5). Other evaluated models (LLAMA-2-7B-Chat, Mistral-7B-Instruct-v0.2, Qwen2.5-7B-Instruct, Phi-4-14B-Instruct) lack equivalent visualizations despite DEEPALIGN being tested on them.
AI-text detection (Finding 6): Score 0.35/1.0 — borderline, not actionable on its own for NDSS-grade prose that has been iteratively revised.

Evidence highlights

Table I (Page 8): DEEPALIGN row shows 45 data points, the vast majority at 0% or 0.2%–0.4%.
Table II (Page 9): AlpacaEval, HumanEval, GSM8k, ARC-Challenge columns; column-difference regularity flagged on 1 column pair.
Section V-B, Page 8: Verbatim statement — "we excluded results for LLAMA-3, Qwen, and Phi4 because the released code does not natively support these architectures."
Figure 2 (Page 3): "Test accuracy of linear classifiers of benign and harmful hidden states across response tokens for each layer" with 4 layer-grouped line sets.
Figure 4 (Page 10): GCG/DRA/Code Attack comparisons, llama3 vs. llama3-ours, showing curves rising from ~0.5 to ~0.9.
Figure 5 (Page 11): DeepSeek-R1-Distill-Qwen-7B under CodeAttack.
Forensic statistics: χ² = 243.1 (p = 0.0000), odd/even = 167/749 = 0.22 (p = 0.0000), Benford deviation at digit positions 1, 3, 4.
DOI: 10.14722/ndss.2026.240004

Notes

Benford and last-digit tests are not reliable on bounded percentages [0, 100] with heavy zero-mass concentration; conclusions from such tests on Table I are not robust.
Vector-rendered scientific charts (PDF line plots) frequently produce false-positive PRNU/copy-move detections; horizontal displacements around gridlines are not diagnostic.
The "too uniformly near zero" pattern is method-consistent with strong hidden-state alignment, but unusual for adversarial-defense benchmarks; raw per-query judge labels are needed.
No image fabrication vectors (e.g., Western blots, gels) are present, as expected for an NLP security paper; suspicion centers on numerical regularity.
Recommended actions: (1) request raw evaluation logs and judge outputs for Tables I–III; (2) request DEEPALIGN training loss curves and random seeds; (3) request clarification on CAT baseline asymmetry; (4) request underlying discriminability data for the 3 models not shown in Figures 2/5.