Ask ChatGPT to recite a passage from Harry Potter, and it might actually do it. But does that mean it will casually leak J.K. Rowling's original text in ordinary conversation?
Researchers at the University of Southern Denmark pose a seemingly simple but long-overlooked question: an LLM being able to leak training data and it actually leaking training data are two entirely different things. Their PropMe framework systematically separates these two notions for the first time, reaching a reassuring conclusion — while models can be forced to emit training data, they almost never do so spontaneously in everyday use.
Capability vs. Propensity: An Overlooked Distinction
Most existing memorization research does the same thing: attack the model. Give it a prefix of some training data (prefix attack) and see whether it can complete the text. This measures memorization capability — the maximum a model can leak under the strongest attack.
It's like testing a safe: you smash it open with a sledgehammer and declare it insecure. In daily use, nobody attacks your safe with a sledgehammer. What you care about is whether the safe pops open on its own during normal use.
That is memorization propensity — the probability that a model actively reproduces training data in non-adversarial, everyday scenarios.
The PropMe Framework: A Three-Tier Evaluation
PropMe spans a continuum from "propensity" to "capability":
Tier 1: Generic prompts. Ask an ordinary question (e.g., "Please explain the basic principles of quantum mechanics") and check whether the answer contains verbatim reproductions of training data. This measures the most natural leakage propensity.
Tier 2: Dataset-specific prompts. Give a prompt thematically related to a training dataset, e.g., "Please discuss existentialist themes in Danish literature" (if the training data contains Danish literature). This measures topic-induced leakage propensity.
Tier 3: Prefix attacks. Provide the start of a training document and ask the model to continue — the traditional memorization capability test.
The researchers also built SimpleTrace, a lightweight tracing tool based on infini-gram, which maps every span of generated text precisely back to its original source document in the training corpus — far more accurate than earlier web-snippet comparison methods.
Key Finding: Can Leak ≠ Will Leak
Experiments ran on two fully open models: Comma (English) and DFM Decoder (a Danish model further pretrained from Comma), evaluated on two datasets (Common Pile and Dynaword).
Finding 1: Memorization under prefix attacks far exceeds non-adversarial settings. Not surprising, but the quantified gap is striking. Prefix attacks extract large amounts of training data, while leakage under generic prompts is nearly undetectable.
Finding 2: Propensity scores are low overall. Under both generic and dataset-specific prompts, memorization propensity is very low. Models do not spontaneously reproduce training data in everyday dialogue.
Finding 3: Continued pretraining reduces memorization of older data. DFM Decoder was further trained on Danish data, and its memorization capability *and* propensity for Common Pile (English data) both dropped. Intuitively, new data partially "overwrites" memory traces of old data.
Finding 4: Comma shows non-negligible leakage propensity in certain settings. Although overall propensity is low, under some specific prompts Comma does reproduce training data. So "models don't leak" cannot be claimed across the board.
Why Does the Distinction Matter?
Legal compliance. GDPR requires organizations to assess risks of personal data disclosure, and the EU AI Act requires cybersecurity assessments for high-risk AI systems. Reporting only "the model can leak X% under the strongest attack" overstates real-world risk; reporting only "the model rarely leaks in daily use" understates the worst case. PropMe reports both, providing a more complete risk picture.
Security auditing. Memorization capability tells you "how bad can it get"; propensity tells you "how safe is it day-to-day." You need both.
Model improvement. A model with high capability but low propensity stores training data internally, yet its generation strategy successfully avoids leaking it — a direction for designing safer models.
Methodological Contribution: Propensity Metric Transformation
PropMe offers more than an evaluation framework: a general metric transformation method. Any existing memorization metric (verbatim length, LCS, near-exact recall, etc.) can be converted into a corresponding propensity metric via a mathematical transform. Researchers don't need new metrics — just apply the transformation to existing ones to measure both capability and propensity.
Limitations and Future Work
PropMe has only been validated on two open models; closed models (GPT, Claude, etc.) cannot use SimpleTrace since their training data is unavailable. Also, the design space of non-adversarial prompts is vast, and the current three-tier evaluation may not cover all possible leakage scenarios.
Still, this work opens an important research direction: AI safety evaluation should focus not only on worst cases, but also on everyday risk. The capability/propensity distinction applies to memorization and to AI safety more broadly — a model being able to do something dangerous doesn't mean it will do it in normal use.
---
Paper: LLMs Can Leak Training Data But Do They Want To?
Code: github.com/N-essuno/PropMe
Authors: Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech (University of Southern Denmark)