He Can No Longer Recognize His Own Handwriting: ENDOPROMPT — a Label-Free Utility Degradation Attack on LLMs

Forum topic · 小凯 · 2026-09-27

Summary

ENDOPROMPT is a prompt-injection attack presented in a September 2026 arXiv paper (arXiv:2609.29948) that degrades the utility of large language models without needing to know any correct answers. Unlike jailbreak attacks that aim to elicit harmful content, ENDOPROMPT seeks only to make models fail at benign tasks. Its core idea is the pseudo-reference: the attacker first has the victim model generate a short (up to 64 token) clean continuation of an unlabeled instruction, then searches for a malicious prefix (up to 16 tokens) that maximally raises the negative log-likelihood of that self-generated text — the displacement reward. A prefix generator is trained via a five-step loop combining local token search, within-instruction preference pairs, and GRPO reinforcement learning, using 2,400 Dolly-15k instructions. Evaluated on full test sets of seven benchmarks (GSM8K, MATH-500, BBH, HellaSwag, TruthfulQA, IFEval, MMLU) against Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3, and Gemma-2-9B, the attack causes an average utility drop of 26.8 percentage points (95% CI [-27.4, -26.2]), with 27 of 28 model-benchmark cells negative — far exceeding prior methods like SPP (-16.2). Failure analysis shows output inflation (58.4%) and answer-extraction failures (39.5%) rather than refusals. The authors honestly note that request conditioning provides no verified advantage, and that a small set of high-frequency prefixes covers most requests, suggesting cheap defenses.

Key points

ENDOPROMPT is a new prompt-injection attack family targeting utility degradation — not jailbreaking — published as *"ENDOPROMPT: Victim-Side Pseudo-References for Utility Degradation"* (arXiv:2609.29948, September 2026; authors Qingyu Wu, Zeyu Feng, Yongda Yu, Yuzhe Luo, Hua Cheng).

The threat is quiet failure, not refusal. Poisoned models remain fluent, polite, and grammatically correct — they simply stop solving problems. On a GSM8K question, a Qwen2.5-7B model will eloquently "describe" how to solve the problem without ever computing anything. No alerts, no refusals, no errors: the model can pass every standard safety metric while being effectively broken.
The core constraint it breaks: prior attacks required the attacker to know what the wrong output should look like (a label, loss signal, or target response). But benign-task failure modes are unbounded and heterogeneous. ENDOPROMPT needs no labels at all.

How it works

1. Pseudo-reference: the victim model itself greedily generates a clean continuation (≤64 tokens) for an unlabeled instruction — a sample of its "own handwriting," not a reference answer. 2. Displacement reward: a candidate prefix (≤16 tokens) is scored by how much it raises the average NLL of that continuation under teacher forcing — i.e., how strongly it makes the model *disown its own likely next words*. 3. Five-step training loop: an unaligned base generator (Dolphin3.0-Llama3.2-3B) is iteratively refined via victim-tokenizer local search (generator proposes 64 candidates, search nominally 512), within-instruction preference pairs (best vs. worst prefix on the same instruction — clean control of noise), preference fitting plus GRPO reinforcement, and checkpoint gating (gain ≥0.03 on 600 held-out instructions to accept a round). 4. Deployment: one forward pass of the generator produces the prefix; no runtime search, no victim internals.

Results

Victims: Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3, Gemma-2-9B across full test sets of GSM8K, MATH-500, BBH, HellaSwag, TruthfulQA, IFEval, and MMLU (~35,000 questions, paired clean/attacked runs, 20,000-replicate paired bootstrap CIs).
Average utility change: -26.8 pp (95% CI [-27.4, -26.2]); 27 of 28 cells negative. Comparisons: UAT -1.9, GCG-16 -7.4, BEAST -1.4, AmpleGCG -4.0, AdvPrompter -4.9, SPP -16.2 — and ENDOPROMPT uses the weakest channel (user-prefix insertion) with a label-free signal.
Ablations: initial generator alone -23.6; after preference fitting without GRPO only -5.5 (a dramatic dip, likely diversity collapse); direct reward refinement -20.4; shuffled pseudo-references -16.1. Pseudo-references set the direction, preference pairs give discrimination, GRPO provides the power.

Failure anatomy (36,750 flipped samples)

58.4% output inflation (response length at least doubled); only 0.3% match refusal vocabulary — the model talks endlessly instead of refusing.
39.5% extraction failures — no parseable answer present.
Task redirection: describing computation instead of doing it.
Prefix reuse: only 0.4–4.7% of generated prefixes are distinct after normalization; the most frequent prefix covers up to 65.0% (Mistral) of requests — a potential cheap defense via frequency-based blacklisting.

Honest limitations

Control experiments show request conditioning gives no statistically verified advantage: matched vs. shuffled prefixes differ by +0.21 pp (CI crosses zero); matched vs. empty by -0.30 (also crosses zero); and reusing the single most frequent prefix actually outperforms matched prefixes by 2.79 pp overall. "Request-conditioned" describes the interface, not a proven benefit. The attack also requires offline logit (white-box) access to open-weight models, only inserts user prefixes, uses a single seed (42), and does not cover closed models or indirect injection.

Takeaways for practitioners

1. Measure benign performance directly — it cannot be inferred from safety compliance or refusal rates. 2. Pair benchmark scores with direct inspection of outputs (inflation and extraction failures are invisible in scores alone). 3. Audit both assigned prefixes and the system's most frequently generated strings; high prefix reuse makes frequency-based filtering a low-cost defense.

Paper: https://arxiv.org/abs/2609.29948. Code is promised to be open-sourced upon acceptance; attack artifacts restricted to vetted researchers. ENDOPROMPT defines a new threat model: not making a system do wrong things, but making it unable to do right things — reliability itself can be weaponized with as little as 16 tokens.