Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models
- Authors: Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron
- arXiv ID: 2609.05381
- Published: 2026-09-04
- Field: AI/ML (molecular property prediction)
- The paper reveals an unsettling fact: when large language models (LLMs) are evaluated on molecular property benchmarks, more than 50% of "predictions" may not be genuine predictions but verbatim retrieval of published values from training data—like a student reciting answer keys rather than solving problems.
- 22 frontier models were audited, including GPT-4, Claude, Gemini, and Llama.
- 12 regression benchmarks were covered, spanning molecular energies, solubility, and toxicity (e.g., ESOL, FreeSolv, Lipophilicity, QM7, QM8).
- On 5 of 12 datasets, over 50% of models exhibited verbatim retrieval.
- Raising the reasoning level increased retrieval behavior by 89%.
- Even when molecular representations were transformed (e.g., SMILES string randomization), the strongest models could still recognize the original labels.
- Verbatim retrieval is widespread: On 5 of 12 benchmarks, more than half the models showed verbatim retrieval behavior.
- Hotspots vs. safe zones: Quantum-mechanics datasets (QM7, QM8) and solubility datasets (ESOL) are heavily affected—these values are precisely computed or measured and widely cited in literature. Datasets involving complex biological activity or toxicity are relatively safer, since values vary and lack a single "standard answer."
- The reasoning paradox: Increasing reasoning level increased verbatim retrieval by 89%. Rather than analyzing more deeply, models may use extended "thinking" time to search memory for matches—raising deep questions about whether chain-of-thought "reasoning" is sometimes disguised retrieval.
- Stubborn memory in strong models: Simple transformations broke retrieval for some models, but the strongest models still recovered original labels even under complex transformations, indicating multi-modal memory associations.
- Busch, M., Tacke, M., Lamaka, S. V., Zheludkevich, M. L., & Cyron, C. J. (2026). *Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models*. arXiv preprint arXiv:2609.05381.
- Related benchmarks: ESOL, FreeSolv, Lipophilicity, QM7, QM8
- Models audited: GPT-4, Claude, Gemini, Llama, and 18 other frontier LLMs
Key Points
The Core Problem
Traditional evaluation metrics such as MAE or RMSE only measure how close predictions are to ground truth—not how the prediction was derived. If a model has already "seen" a molecule and its published value during training, its low error may reflect memorization of training data, not genuine generalization.
Detection Methodology
Because most frontier models are black-box APIs, the authors designed a clever detection scheme:
1. Six-digit fingerprint matching: If a model outputs a value matching a literature value to six significant figures, the probability of coincidence is extremely low (~one in a million), so this is treated as high-confidence evidence of retrieval rather than prediction. 2. Transformation attacks: The authors applied SMILES randomization, graph isomorphism (atom renumbering), and mixed representation formats. If a model outputs different values under transformed representations, it relied on specific memory cues; consistent output suggests actual computation.
Findings
Implications
The paper exposes a systemic flaw in AI evaluation: with trillions of training tokens, we cannot fully know what a model has "seen," and even "new" test sets may closely resemble training data. High scores may reflect exposure volume rather than understanding.
It also touches a deeper tension: for engineering applications (e.g., rapid drug screening), memorization may be no worse than understanding if answers are correct; but for scientific discovery aimed at underlying principles, memorization is worthless.
Possible Mitigations
1. Dynamic benchmarks: Generate test data on the fly via chemical simulations—fresh data cannot be in training corpora, though this is costly. 2. Adversarial transformations: Require consistent answers across randomized representations. 3. Reasoning audits: Ask models for derivations—but this circles back to the chain-of-thought paradox. 4. Decontamination: Ensure test data is excluded from training—nearly impossible at scale, and "similar but not identical" data still leaks indirect cues.
Conclusion
The paper is a mirror reflecting a blind spot in current AI evaluation: high scores do not equal high ability, and low error does not equal understanding. Its value lies not only in exposing memorization in molecular property prediction, but in posing a universal question: how do we distinguish AI's "memory" from genuine "understanding"?