Paper Overview
Field: NLP Authors: Sarah Wyer, Sue Black, Noura Al Moubayed Published: 2026-09-17 arXiv: 2609.20779
Summary
Safety evaluations for large language models rely on surface-form classifiers that report declining harm scores across model generations. This paper provides evidence that this methodology is systematically incomplete: explicit discriminatory content is transformed rather than removed — a phenomenon the authors call harm laundering.
Key Findings
- Scale: 450,000 gender-directed completions analyzed across 15 models spanning GPT-2 through GPT-5 (OpenAI GPT lineage; three demographic conditions).
- Sexual violence clusters prevalent in GPT-2 women-directed output disappear by GPT-4.
- Men-directed completions gain positive representational territory (caregiving, emotional range, ally identity) that women-directed completions do not.
- The pattern is most visible at GPT-5: Topic 5 (1,997 documents) frames breast cancer as a men's rights debate, while women-directed output shows zero equivalent topic clusters. Three independent classifiers scored this content as non-toxic.
- Sentiment scores invert at GPT-4: earlier models demeaned women; later models overcorrect.
- At the GPT-4 alignment boundary, topic diversity for women-directed completions dropped 36% relative to men-directed ones (W/M = 0.58, compared to 0.91 in GPT-2).
- REGARD representational harm differences correlate with release date (rho = +0.55, p = .034), while Detoxify toxicity does not (rho = -0.23, p = .42): toxicity scores decline while representational harm grows.
- A formal three-criterion test for harm laundering.
- A three-stage detection protocol applicable to any generative model.
- The conclusion that, within the OpenAI GPT lineage, declining toxicity scores are not a sufficient proxy for declining harm.