LLMSurgeon: Inferring LLM Pretraining Data Mixtures from Model Outputs Alone

Paper: LLMSurgeon: Diagnosing Data Mixture of Large Language Models (arXiv:2605.30348) Authors: Yaxin Luo, Jiacheng Cui, Xiaohan Zhao, Xinyi Shang, Jiacheng Liu, Xinyue Bi, Zhaoyi Li, Zhiqiang Shen Institutions: VILA Lab, MBZUAI; UCL Core claim: The domain-level distribution of an LLM's pretraining corpus can be recovered from the model's generated text alone — no access to training data, weights, or internal states required.

The Problem: Digital DNA Nobody Discloses

Modern LLM training mixes trillions of tokens from web pages, books, papers, code, and forums. These mixture proportions form a model's "digital DNA," shaping its strengths, biases, and failure modes — yet they are almost never disclosed. Commercial labs keep them secret, and only a handful of open models (OLMo, Amber) publish full recipes.

Prior probing tools rely on membership inference attacks (MIA), which ask whether a specific text was seen during training. But aggregating millions of point-level predictions into a global mixture estimate fails mathematically: MIA errors are biased, sparse, and domain-dependent (code is memorized more than generic web text), computationally prohibitive, and poorly defined for long documents. LLMSurgeon instead aims to map the whole beach, not find a single grain.

Method: A Label-Shift Inverse Problem

LLMSurgeon assumes label shift: generation changes only how often each domain appears, not each domain's conditional language statistics. The recipe may shift from 20% code in training to 10% code in generation, but the code the model does produce remains statistically faithful to training code. Under this assumption, the expected classifier output over generated samples is Cᵀπ, where C is a soft confusion matrix and π the true mixture — so recovering π is a constrained least-squares inverse problem.

Three stages:

1. Train a domain classifier and measure its bias. An external classifier (fine-tuned DistilBERT works best) assigns texts to predefined domains. Its systematic errors are captured in a soft confusion matrix C, where C_ij = E_{x~p_i}[f_φ(x)_j] — expected probabilities, not hard labels. Keeping the classifier's "hesitation" is key to stabilizing the inverse problem.

2. Sample the target model with neutral prompts. Prompts like "please continue this text" avoid style forcing. Ablations show this matters enormously: instructive prompts crashed OLMo-1B's estimation accuracy from 94.46% to 22.71%. The generated corpus is scored by the classifier to yield a biased observed distribution p̄.

3. Solve the inverse problem (de-blur). Estimate π̂ = argmin_{π∈Δ^{K−1}} ‖Cᵀπ − p̄‖₂² via standard convex optimization — knowing how blurry the glass is, reconstruct the original image.

LLMScan Benchmark and Results

Since closed-model recipes are unknown, the authors built LLMScan: 8 fully open models (1B–65B parameters) with officially documented mixtures, at three granularities:

• Coarse (K=6): web, code, wiki, books, papers, forums — LLaMA-1, OLMo, Amber. LLMSurgeon reaches 95.14% overlap accuracy on LLaMA-1-7B (vs. 35–48% for MIA baselines; 93.42% without inverse correction).
• Medium (K=17): The Pile categories — Pythia, GPT-Neo. LLMSurgeon: 63.20% on Pythia-12B vs. MIA baselines stuck near 52–56%.
• Fine (K=87): The Stack programming languages — StarCoder. Absolute accuracy drops to 30.37% (best baseline: 27.54%). The bottleneck is ill-conditioning: near-synonymous domains make C nearly singular (R² falls from 0.99 coarse to 0.01 fine).

Notable Ablation Findings

• Prompt style is critical. Neutral prompts dominate; leading questions distort the observed "diet."
• Mixtures drift during training. Checkpoint tracking on Amber-13B shows early oscillation (possible curriculum/staged data injection); OLMo-1B is stable. LLMSurgeon can thus serve as a training-stability monitor.
• Reference-data sweet spot: 5,000 documents per domain; 10,000 slightly hurts.
• Backbone choice matters: DistilBERT beats TF-IDF, MLP, and from-scratch Transformers by up to 4.92 points.

Safety Audit Demo

In controlled GPT-2 experiments where 5%, 10%, and 20% of training tokens were replaced with toxic content (RealToxicityPrompts), LLMSurgeon estimated 7.90%, 12.00%, and 22.73% respectively — monotone and accurate enough for low-cost, front-line screening before expensive red-teaming. It provides macro-level warning, not forensic attribution.

Honest Limitations

• Alignment interference: heavy RLHF/instruction tuning may distort the pretraining prior that neutral prompts are meant to reveal — likely a harder problem than the paper suggests.
• Closed-world assumption: unseen domains get forced into the K known categories; the method cannot discover new domains.
• Information-theoretic limits: semantically indistinguishable domains make the inverse problem unstable regardless of classifier quality.
• Cross-version generalization is promising but unproven: hyperparameters from OLMo-1/2 transferred to OLMo-3 with 86.41% accuracy; distribution-shifted families remain untested.
• Open question: effectiveness on heavily post-trained commercial models (GPT-4o, Claude-class) is unknown.

Why It Matters

LLMSurgeon enables auditing without keys: no weights, no logs, no cooperation from the vendor — just queries and analysis. As the EU AI Act and NIST frameworks demand verifiable AI transparency, this offers regulators, researchers, and journalists a path to check models' "lineage claims" externally. Extensions to multimodal models could even reshape copyright litigation, letting plaintiffs estimate the proportion of protected content in training data from generations alone — though the image-domain inverse problem remains open.

References 1. Luo et al., "LLMSurgeon: Diagnosing Data Mixture of Large Language Models", arXiv:2605.30348, 2026. 2. Groeneveld et al., "OLMo: Accelerating the Science of Language Models", 2024. 3. Biderman et al., "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling", ICML 2023. 4. Gao et al., "The Pile: An 800GB Dataset of Diverse Text for Diverse Text for Language Modeling", 2020. 5. Shen et al., "SlimPajama-DC: A Data-Centric Approach for Efficient and Effective LLM Pre-training", 2023.