Academic Integrity Report: 'The Asymmetric Vulnerability: Bypassing LLM Defenses via Guardrail-Model Mismatch' (DOI: 10.1145/3774904.3792438)

Academic fraud report · Geng Detector

Summary

This report assesses potential data fabrication in a WWW '26 paper on bypassing LLM safety guardrails. Two decisive anomalies were identified in Table 1 (Page 6). First, FlipAttack baseline ASR values are identical across three architecturally distinct models (GPT-4o, DeepSeek-r1, Gemini-2.5): 3.5 under Llama Guard, 0.5 under WildGuard, and 5.5 under GuardReasoner, an event whose probability is vanishingly small given independent inference pipelines. Second, for original AdvBench prompts, detection rates are identical across no-guardrail, ShieldGemma, and OpenAI Moderation conditions for each model (GPT-4o: 1.0/1.0/1.0; DeepSeek-r1: 3.5/3.5/3.5; Gemini-2.5: 4.5/4.5/4.5), implying the guardrails are inert, which contradicts published baselines. These patterns strongly suggest copy-pasted synthetic data. Secondary concerns include an implausibly high GPT-4o/human Pearson correlation (0.989) and reproducibility risks from unverified 2025-2026 model versions. Confidence in findings 1-2 is high; the authors should release raw API logs.

Verdict

🔴 Confirmed fabrication indicators (high confidence). Table 1 contains two independent statistical impossibilities indicating copy-pasted/fabricated baseline data. Secondary issues (judge-model correlation, model-version reproducibility) warrant further investigation but are not independently conclusive.

Key findings

Finding 1 (Fatal): FlipAttack ASR values are bit-for-bit identical across GPT-4o, DeepSeek-r1, and Gemini-2.5 (LG=3.5, WG=0.5, GR=5.5). With N=200 per cell, the probability of three architecturally independent models yielding identical attack success rates to one decimal place is effectively zero. This pattern is consistent with deliberate fabrication rather than coincidence.
Finding 2 (Fatal): Original AdvBench detection rates are identical across no-guardrail (w/o), ShieldGemma (SG), and OpenAI Moderation (OM) for every model tested (GPT-4o: 1.0/1.0/1.0; DeepSeek-r1: 3.5/3.5/3.5; Gemini-2.5: 4.5/4.5/4.5). Functionally, this asserts that ShieldGemma and OpenAI Moderation provide zero additional protection over no guardrail at all on canonical harmful prompts, which contradicts widely published benchmarks.
Finding 3 (Concerning): Appendix D, Table 4 reports a GPT-4o-vs-human Pearson correlation of ρ=0.989 for attack scoring, exceeding typical inter-annotator agreement ranges (0.8–0.9) and raising concerns of prompt leakage, score truncation, or selective curation.
Finding 4 (Reproducibility): Reliance on cutting-edge or lightly documented targets (GPT-o3, Qwen3-8B-abliterated, Gemini-2.5) for black-box evaluation, while chronologically consistent with a 2026 publication, increases the risk that results cannot be independently reproduced.

Evidence highlights

DOI: 10.1145/3774904.3792438
Location of primary evidence: Table 1, Page 6
FlipAttack row (verbatim reported values): GPT-4o (LG=3.5, WG=0.5, GR=5.5); DeepSeek-r1 (LG=3.5, WG=0.5, GR=5.5); Gemini-2.5 (LG=3.5, WG=0.5, GR=5.5)
Origin AdvBench row (verbatim reported values): GPT-4o (w/o=1.0, SG=1.0, OM=1.0); DeepSeek-r1 (w/o=3.5, SG=3.5, OM=3.5); Gemini-2.5 (w/o=4.5, SG=4.5, OM=4.5)
Implied sample size: N=200 per cell (inferred from report context)
Secondary metric: GPT-4o-as-judge vs. human Pearson ρ=0.989 (Appendix D, Table 4, Page 10)

Notes

Findings 1 and 2 are based on the final-version PDF; the report should be re-verified against the camera-ready if it differs.
The detection was performed using text-and-table analysis only; figure-level image forensics were not applicable to the artifacts available.
Authors have not yet (to our knowledge) been contacted; raw API logs and inference seeds are the decisive artifact to request.
Confidence in Findings 1–2 is high. Findings 3–4 are suggestive but require author-side data for confirmation.
This report is an AI-assisted screening output and does not constitute a formal finding of misconduct; institutional investigation is required for any official determination.