Has Your LLM Really Memorized That Article? Why Black-Box Membership Inference Attacks Are Failing Across the Board

Imagine this scenario: you are a copyright lawyer representing an author suing an AI company. Your core argument: their model secretly "memorized" your client's article during training. The judge turns to you and asks a fatal question—

"Do you have evidence?"

You present the model's generated text and compare it word by word with the original—it does look very similar. But the opposing lawyer stands up and counters: "That's not memorization, that's inference. The model simply deduced the correct answer from context, like a smart student who doesn't need to memorize the textbook to answer questions correctly."

How do you distinguish "memorized" from "inferred"?

This is not a fictional courtroom drama. It is one of the most difficult problems in AI today—Data Contamination Detection. A recent paper from the University of Amsterdam and Elsevier delivers a troubling answer: under black-box conditions, none of our current methods can reliably answer this question.

What Is a Membership Inference Attack?

A Membership Inference Attack (MIA) has a simple goal: given a piece of text and a large language model, determine whether that text appeared in the model's training data. It's like asking: "Has this student memorized this question?"

Methods fall into three categories, by increasing level of access:

• White-box: You can see all model parameters (weights). Most accurate, but almost no commercial model grants this access.
• Gray-box: You can see the model's output probability distribution (logits). High accuracy, but requires special API-level permissions.
• Black-box: You only see the model's plain-text output. Most practical, but also hardest.

This paper focuses on black-box methods—the only option for closed-source models like GPT-4 and Claude.

Four "Interrogation Techniques," All Failed

The paper systematically evaluated four black-box MIA methods across six mainstream LLMs. The results were discouraging.

1. Name Cloze Queries

Principle: Mask proper nouns (names of people and places) in the text and have the model fill in the blanks. Correct answers suggest memorization.

Problem: The model may be inferring rather than recalling. If the context already mentions certain clues, filling in the answer requires no memorization of the original text.

2. DE-COP (Multiple-Choice Paraphrase Detection)

Principle: Give the model one original passage and three paraphrased versions, and ask it to pick the "original."

Problem: Differences between paraphrases and the original may be so obvious that the model can distinguish them through semantic understanding alone, without memory.

3. Prefix Probing

Principle: Give the model the first half of an article and ask it to continue the rest, then compare with the original.

Problem: Modern models are too smart—they refuse to reproduce text verbatim. The paper found GPT-4o and GPT-4o-mini outright refused the original prompt, offering "I can give you a summary" instead. Researchers had to use jailbreak prompts to get results—evidence in itself that models have learned "anti-recitation" behavior.

4. Familiarity Ranking — The Paper's New Method

Principle: Give the model three passages—original, paraphrased, and random—and ask it to rank them by "likelihood of appearing in a particular article." Correct ranking suggests familiarity with the original.

Innovation: Unlike crude binary classification (memorized / not memorized), ranking lets the model express intermediate states like "this seems familiar but I'm not sure."

The Data: What AUC-ROC ≈ 0.5 Means

The paper tested all methods on two datasets (results on the arXiv dataset):

| Method | GPT-4o | Claude 3.5 | GPT-3.5 | |--------|--------|------------|---------| | DE-COP | 0.529 | 0.548 | 0.482 | | Familiarity Ranking | 0.568 | 0.563 | 0.487 | | Name Cloze | 0.556 | 0.519 | 0.548 | | Prefix Probing | 0.500 | 0.500 | 0.500 |

An AUC-ROC of 0.5 means pure chance—a coin flip. The best result, 0.568, is far from "reliable detection" (typically >0.7).

TPR (true positive rate) vs. FPR (false positive rate) is even more revealing:

• Claude 3.5 Sonnet under DE-COP: TPR 0.918, but FPR also 0.823
• The model answers correctly on unseen texts 82% of the time—it isn't "reciting," it's "reasoning"

Core finding: the more advanced the model, the stronger its reasoning, and the blurrier the line between memory and inference.

Why Is This Happening? Three Deep Reasons

1. Models Are Too Smart

GPT-4o and Claude 3.5 have powerful in-context reasoning. Given a title and a passage, they can judge "this text does look like it comes from that paper" through semantic understanding, without ever having memorized it. Like a well-read scholar who can identify "this reads like the author's style" without memorizing every essay.

2. Models Have Learned "Anti-Recitation"

The paper observed an interesting phenomenon: newer models actively refuse verbatim reproduction. GPT-4o responds to continuation requests with "I can give you a summary." Researchers had to use jailbreak prompts to bypass this. AI companies have evidently built copyright protection mechanisms into their models.

3. Memorization Is a Spectrum

The paper distinguishes three kinds of "memory":

1. Verbatim memorization: reproducing text word for word 2. Factual memorization: remembering specific facts 3. Conceptual memorization: remembering core ideas and expressing them differently

Black-box methods can only detect the first kind, but modern models mostly exhibit the second and third—and the boundaries between them are inherently blurry.

What Does This Mean?

For AI Practitioners

If you do model evaluation, don't blindly trust "my benchmark was never seen by the model." Even if test data truly isn't in the training set, the model may reach correct answers through reasoning, inflating apparent generalization.

For Copyright

Proving via black-box methods that "the model memorized my article" is currently nearly impossible. This has direct implications for cases like *The New York Times v. OpenAI*—plaintiffs will need stronger evidence.

For AI Safety Research

Black-box MIA may simply be the wrong direction. Gray-box methods (like EM-MIA) require more access but achieve near-perfect accuracy; white-box methods are more direct. The problem: the most advanced models (GPT-4, Claude) offer neither.

Personal Reflection

The paper's conclusion seems pessimistic—"black-box methods don't work"—but it reveals something deeper:

We may need to redefine "memorization."

When a model can infer an article's content from its title, is that "memory" or "understanding"? If a human reads a book and later restates its core ideas in their own words, we say they "remembered" it. Why do we demand verbatim reproduction before calling AI behavior "memorization"?

One detail stuck with me: in the Familiarity Ranking experiments, models gave high scores (8–10) to both the original and paraphrased texts, and low scores (0–3) to random texts. The models can clearly distinguish "relevant" from "irrelevant"—they just can't distinguish "relevant because memorized" from "relevant because inferred."

This may be the most astonishing and unsettling thing about LLMs: when reasoning becomes powerful enough, memorization becomes undetectable.

---

Paper Information

• Title: Detecting Data Contamination in Large Language Models
• Authors: Juliusz Janicki (University of Amsterdam), Savvas Chamezopoulos (Elsevier), Evangelos Kanoulas (University of Amsterdam), Georgios Tsatsaronis (Elsevier)
• Published: April 21, 2026
• arXiv: 2604.19561
• HTML version: arxiv.org/html/2604.19561
• Open-source code: No public repository yet