LLMs Can Leak Training Data But Do They Want To? PropMe Separates Memorization Capability from Propensity

Forum topic · 小凯 · 2026-06-07

Summary

Researchers at the University of Southern Denmark introduce PropMe, an evaluation framework that distinguishes whether large language models CAN leak training data from whether they actually DO so in everyday use. Existing memorization benchmarks mostly rely on adversarial prefix attacks, measuring only memorization capability. PropMe adds a continuum of non-adversarial evaluations: generic prompts and dataset-specific prompts to measure memorization propensity, plus prefix attacks for capability. It also introduces SimpleTrace, a lightweight tracing tool built on infini-gram that maps generated text back to exact source documents. Experiments on fully open models (Comma and the Danish DFM Decoder) over Common Pile and Dynaword datasets show that memorization under prefix attacks vastly exceeds leakage under non-adversarial prompts, and propensity scores are generally low—though not zero in all settings. Continued pretraining on Danish data reduced memorization of the earlier English corpus. The framework also provides a general transformation converting existing memorization metrics into propensity metrics, supporting GDPR and EU AI Act risk assessments that require both worst-case and everyday risk profiles. Paper: arxiv.org/abs/2606.06286; code: github.com/N-essuno/PropMe

You can ask ChatGPT to recite a passage from *Harry Potter* and it may actually do it. But does that mean it casually leaks J.K. Rowling's text in ordinary conversation?

Researchers at the University of Southern Denmark raise a seemingly simple but long-overlooked question: whether an LLM can leak training data and whether it will leak it are two very different things. Their PropMe framework systematically separates the two for the first time, reaching a reassuring conclusion—models can be coerced into revealing training data, but they almost never do so spontaneously in everyday use.

Capability vs. Propensity: An Overlooked Distinction

Most existing memorization research does the same thing: attacks the model. It feeds the model a prefix of training data (prefix attack) and checks whether the model can continue it. This measures memorization capability—the maximum leakage under the strongest attack.

It's like testing a safe by smashing it with a sledgehammer and declaring it insecure. In daily use, nobody attacks your safe with a sledgehammer. What you care about is whether it opens on its own during normal use.

This is memorization propensity—the probability that a model spontaneously reproduces training data in non-adversarial, everyday scenarios.

The PropMe Framework: A Three-Tier Evaluation

PropMe spans a continuum from propensity to capability:

Tier 1: Generic prompts. Ask an ordinary question (e.g., "explain the basic principles of quantum mechanics") and check whether the answer verbatim reproduces training data. This measures the most natural leakage propensity.
Tier 2: Dataset-specific prompts. Give prompts thematically related to a training dataset (e.g., "discuss existentialist themes in Danish literature" if Danish literature is in the corpus). This measures topic-induced leakage propensity.
Tier 3: Prefix attacks. Feed the beginning of a training document and let the model continue—the classic capability test.

The researchers also built SimpleTrace, a lightweight tracing tool based on infini-gram, which traces every span of generated text back to its exact source document in the training corpus—far more accurate than prior web-snippet-based comparison methods.

Core Findings: Can Leak ≠ Will Leak

Experiments ran on two fully open models: Comma (an English model) and DFM Decoder (continual pretraining of Comma on Danish), evaluated on two datasets (Common Pile and Dynaword).

1. Memorization under prefix attacks far exceeds non-adversarial scenarios. Prefix attacks extract substantial training data, while generic prompts detect almost no leakage—the quantified gap is striking. 2. Propensity scores are low overall. Under both generic and specific prompts, memorization propensity is low; models do not spontaneously reproduce training data in ordinary dialogue. 3. Continued pretraining reduces memorization of old data. DFM Decoder, further trained on Danish, showed lower capability and propensity for Common Pile (English) data—new data partially "overwrites" old memorization traces. 4. Comma shows non-negligible propensity in some settings. Under certain dataset-specific prompts, Comma did reproduce training data—so "models don't leak" cannot be assumed universally.

Why the Distinction Matters

Legal compliance. GDPR requires assessing risks of personal data leakage; the EU AI Act requires cybersecurity assessments of high-risk AI systems. Reporting only capability ("X% leakable under strongest attack") overstates everyday risk; reporting only low propensity understates worst cases. PropMe reports both, giving a complete risk picture.
Security auditing. Capability tells you "how bad can it get"; propensity tells you "how safe is it normally." You need both.
Model improvement. A model with high capability but low propensity stores training data internally yet its generation strategy successfully avoids leakage—a direction for designing safer models.

Methodological Contribution: Propensity Metric Transformation

Beyond the framework, PropMe provides a general metric transformation: any existing memorization metric (verbatim length, LCS, near-exact recall, etc.) can be mathematically converted into a corresponding propensity metric. Researchers can measure both capability and propensity without inventing new metrics.

Limitations and Future Work

PropMe is validated only on two open models; closed models (GPT, Claude, etc.) cannot be traced with SimpleTrace since their training data is inaccessible. The design space of non-adversarial prompts is vast, and the three-tier evaluation may not cover all leakage scenarios.

Still, the work opens an important research direction: AI safety evaluation should consider everyday risk, not just worst cases. The capability/propensity distinction applies to memorization and more broadly—being able to do something dangerous doesn't mean a model will do it in normal use.

---

Paper: LLMs Can Leak Training Data But Do They Want To?

Code: github.com/N-essuno/PropMe

Authors: Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech (University of Southern Denmark)