Molecular Déjà Vu: When AI 'Cheats' on Exams
This post summarizes and analyzes the paper "Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models" by Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, and Christian J. Cyron (arXiv:2609.05381).
Key Points
- When evaluated on molecular property benchmarks, more than 50% of "predictions" from some frontier LLMs may not be genuine predictions, but verbatim retrieval of published literature values memorized during training.
- 22 frontier models were audited (including GPT-4, Claude, Gemini, and Llama families) across 12 regression benchmarks covering molecular energies, solubility, and toxicity.
- On 5 of 12 datasets, over 50% of models exhibited verbatim retrieval.
- Increasing the reasoning level increased retrieval behavior by 89% — more "thinking" appears to give models more opportunity to search their memory rather than reason more deeply.
- Even when molecular representations were transformed (e.g., SMILES randomization), the strongest models still identified the original values, suggesting multi-modal memory associations.
- Hotspots: QM7, QM8 (quantum-chemical energies), and ESOL-style solubility datasets — values are uniquely determined and widely cited, so they easily enter training data.
- Safer zones: datasets involving complex bioactivity or toxicity, where values have high variability and no single "standard answer."
- Dynamic benchmarks: generate test values on the fly via simulation (costly, and simulation accuracy must be validated).
- Adversarial transformations: require consistent answers across randomized input representations.
- Reasoning audits: examine derivations, though models may merely simulate plausible reasoning.
- Decontamination: excluding test data from training is nearly impossible at trillions-of-tokens scale, and near-duplicates still leak indirect clues.
- Busch, M., Tacke, M., Lamaka, S. V., Zheludkevich, M. L., & Cyron, C. J. (2026). *Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models*. arXiv preprint arXiv:2609.05381.
- Benchmarks referenced: ESOL, FreeSolv, Lipophilicity, QM7, QM8.
The Detective Method: Six-Digit Fingerprints
The core detection idea: if a model outputs a value matching a published number in the first six significant digits, the probability of coincidence is roughly one in a million. Such matches can therefore be attributed to retrieval rather than prediction with high confidence — like a student whose exam answer matches the answer key to six decimal places.
Researchers also ran transformation attacks: randomizing SMILES strings, applying graph isomorphism permutations, and mixing representation formats. Weaker models failed to produce exact matches under transformation, but the strongest models recovered the original values regardless of representation.
Where Retrieval Is Worst
Implications
1. Evaluation crisis: Standard MAE/RMSE metrics measure closeness to a reference value, not how the answer was derived. High scores may reflect memorization, not generalization — the equivalent of a student who crammed past exams rather than learned chemistry. 2. The reasoning paradox: If chain-of-thought mainly enables memory lookup, the trustworthiness of a model's "explanations" is questionable. 3. Science vs. engineering: For engineering applications (e.g., quick molecular screening), memorization that yields correct answers may suffice; for scientific discovery of underlying principles, it is worthless.
Possible Mitigations
Conclusion
The paper's central lesson: high scores are not high ability, and low error is not understanding. Distinguishing AI "memory" from "understanding" remains an open problem — but now we at least know the right questions to ask.