Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models
Authors: Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron arXiv ID: 2609.05381 Published: 2026-09-04 Field: AI/ML (molecular property prediction)
Core Finding
When large language models (LLMs) are evaluated on molecular property benchmarks, more than 50% of their "predictions" may not be genuine predictions at all—but verbatim retrieval of published literature values memorized during training. It is like a student who, instead of solving a problem, recites the standard answer.
Key Numbers
- 22 frontier models audited, including GPT-4, Claude, Gemini, and Llama
- 12 regression benchmarks covering molecular energies, solubility, toxicity, and more
- On 5 of 12 datasets, more than 50% of models exhibited verbatim retrieval
- Raising the reasoning level increased retrieval behavior by 89%
- Even after transforming molecular representations (e.g., SMILES string randomization), the strongest models still recognized the original labels
- ESOL — aqueous solubility
- FreeSolv — solvation free energy
- Lipophilicity — lipophilicity
- QM7/QM8 — quantum-mechanical energies
- Verbatim retrieval is widespread but uneven. Hotspots: QM7, QM8, ESOL—datasets with uniquely correct, widely cited values likely present in training corpora. Safer zones: datasets with high intrinsic variability (e.g., complex bioactivity/toxicity), where no single "standard answer" exists.
- The reasoning paradox: increasing reasoning level *increased* verbatim retrieval by 89%. Extended chain-of-thought may give the model more "time" to search memory rather than analyze the problem—raising deep questions about whether CoT explanations reflect reasoning at all.
- Transformation attacks: simpler models broke down under SMILES randomization, but the strongest models recovered the original labels even under complex mixed representations—evidence of multi-representation memory associations.
- Evaluation crisis: mainstream evaluation assumes held-out test sets measure capability, but with trillions of training tokens we cannot know what models have seen, and "new" tests may closely resemble training content.
- Science vs. engineering: for applications like fast drug-candidate screening, memorization that yields correct answers may suffice; for scientific discovery of underlying physicochemical principles, it is worthless.
- Behaviorism vs. cognition: current AI evaluation is almost purely behavioral—judging outputs, not processes. This paper argues the process may matter as much as the result.
- Busch, M., Tacke, M., Lamaka, S. V., Zheludkevich, M. L., & Cyron, C. J. (2026). *Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models.* arXiv preprint arXiv:2609.05381.
- Benchmarks referenced: ESOL, FreeSolv, Lipophilicity, QM7, QM8
- Models involved: GPT-4, Claude, Gemini, Llama, and other frontier LLMs
Background: A Subtle Loophole in Benchmarking
LLMs are increasingly used to design drugs, predict material properties, and optimize reactions. Common benchmarks include:
These are typically regression tasks: given a molecular representation (e.g., a SMILES string), predict a continuous value. Evaluation relies on MAE/RMSE, which measures distance from ground truth but not *how* the answer was derived. If a model already "saw" the molecule and its value during training, low error may reflect memorization, not generalization.
The Detective Method: Six-Digit Fingerprints
Detecting memorization is hard because most frontier models are black-box APIs, molecular representations are diverse, and models may exploit structural similarity. The authors' core idea: if a model outputs a value matching published data to six significant digits, this is unlikely to be coincidence (~one in a million by chance), so it can be flagged as retrieval rather than prediction.
They also ran transformation attacks: SMILES randomization (the same molecule has many valid SMILES), graph isomorphism transforms (reordering atom indices), and mixed representation formats. If outputs stay exactly consistent under transforms, the model may genuinely compute; if only the canonical form yields exact matches, it relies on memorized cues.
Results
Deeper Implications
Mitigation Strategies
1. Dynamic benchmarks — generate test values on the fly via chemical simulation (costly, and simulation accuracy must be validated). 2. Adversarial transformations — require consistent answers across multiple representation transforms. 3. Reasoning-process audits — check whether derivations are substantive or hollow, though CoT may be disguised retrieval. 4. Decontamination — keep test data out of training data, practically near-impossible at scale, and "similar but not identical" data still leaks indirect cues.
Conclusion
High scores do not equal high ability; low error does not equal understanding. When admiring LLM performance in chemistry, physics, and math, we must ask: are these models thinking, or remembering? As a Feynman-style test puts it: if a model cannot answer a problem it has never seen, it has not truly understood. The paper's lasting value is posing the general question: how do we distinguish AI memory from understanding?