Molecular Déjà Vu: When AI 'Predicts' Chemistry by Remembering
*(English adaptation of a Chinese forum post analyzing the paper)*
Paper Info
- Title: Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models
- Authors: Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron
- arXiv ID: 2609.05381
- Published: 2026-09-04
- Field: AI/ML (molecular property prediction)
- 22 frontier models audited, including GPT-4, Claude, and Gemini
- 12 regression benchmarks covering molecular energies, solubility, toxicity, and more
- On 5 datasets, over 50% of models exhibit verbatim retrieval
- Raising the reasoning level increased retrieval behavior by 89%
- Even after transforming molecular representations (e.g., SMILES string randomization), the strongest models still identified the original labels
- ESOL: water solubility
- FreeSolv: hydration free energy
- Lipophilicity: lipophilicity
- QM7/QM8: quantum-mechanical energy values
- Randomized SMILES strings (the same molecule has many valid SMILES forms)
- Graph isomorphism transformations (reordering atom numbering)
- Mixing representation formats
- Worst affected: QM7, QM8 (quantum data), ESOL (solubility) — these have unique "correct" values that are widely cited and easily absorbed into training data
- Relatively safe: complex bioactivity/toxicity datasets — these values are inherently variable with no single standard answer
- Training corpora span trillions of tokens; no one knows what the model has "seen"
- Even "new" test items may closely resemble training content
- For engineering applications (e.g., rapid drug-candidate screening), memorization may be acceptable if answers are correct
- For scientific discovery (understanding physicochemical principles), memorization is worthless
- Busch, M., Tacke, M., Lamaka, S. V., Zheludkevich, M. L., & Cyron, C. J. (2026). Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models. *arXiv preprint arXiv:2609.05381*.
- Related benchmarks: ESOL, FreeSolv, Lipophilicity, QM7, QM8
- Models involved: GPT-4, Claude, Gemini, Llama, and 22 frontier LLMs in total
Core Finding
The paper reveals an uncomfortable truth: when large language models (LLMs) are evaluated on molecular property benchmarks, more than 50% of their "predictions" may not be predictions at all, but verbatim retrieval of published values memorized during training. It is like a student who, instead of solving an exam problem, simply recites the standard answer.
Key Numbers
Background
From Chatbots to Molecular Designers
LLMs are now used to design drugs, predict material properties, and optimize chemical reactions. Benchmarks used to evaluate them include:
These are regression tasks: given a molecular description (e.g., a SMILES string), predict a continuous value.
A Subtle Flaw
Standard metrics like MAE and RMSE only measure the gap between prediction and ground truth—not how the prediction was derived. If the model already "saw" the molecule and its true value during training, low error may reflect memorization, not generalization. The model may be a super-memorizer rather than a real chemist.
How to Catch the "Cheating"
The Six-Digit Fingerprint
The authors' core idea: if a model outputs a value matching the published literature to six significant digits, that is extremely unlikely to be coincidence (roughly a one-in-a-million match for floating-point precision). A six-digit match is therefore high-confidence evidence of retrieval rather than prediction—like a student whose answer matches the key to six decimal places.
Transformation Attacks
To confirm, the researchers applied molecular representation transformations:
If outputs change under transformation, the model relied on specific memorization cues; if outputs stay consistent, it may genuinely be computing.
Results: An Uncomfortable Truth
Verbatim Retrieval Is Widespread
Across 22 models (GPT-4/GPT-3.5, Claude, Gemini, Llama, and others):
> On 5 of 12 benchmarks, more than 50% of models show verbatim retrieval behavior.
Hotspots vs. Safe Zones
The Reasoning Paradox: Smarter Means More Cheating?
Most counterintuitive finding: raising the reasoning level increased verbatim retrieval by 89%. Asking the model to "think step by step" may not deepen analysis—it may give the model more time to search memory for matches. This casts doubt on Chain-of-Thought: if the "reasoning" is disguised retrieval, can we trust the explanations?
Strong Models Resist Transformations
Simple SMILES randomization broke some models' exact matches, but the strongest models still recovered the original labels even under complex transformations, suggesting they built multi-view memory associations—not just the answer, but every phrasing of the question.
Deeper Implications
An Evaluation Crisis
The mainstream evaluation paradigm (train on big data, test on a held-out split) breaks down when:
High scores may measure study volume, not understanding.
Science vs. Engineering
Behaviorism vs. Cognitivism
Current AI evaluation is almost purely behaviorist—only outputs matter. This paper argues the process may matter as much as the result.
Mitigation Strategies
1. Dynamic benchmarks: generate test values on-the-fly via chemical simulation (expensive, and simulation accuracy itself needs validation) 2. Adversarial transformations: require consistent answers across multiple representation variants; failure indicates reliance on memorization cues 3. Reasoning-process audits: demand derivations, not just answers—though models may fabricate plausible-sounding reasoning 4. Decontamination: exclude test data from training—nearly impossible at scale, and "similar but not identical" data still leaks indirect clues
Conclusion
> High scores do not equal high ability; low error does not equal understanding.
As we marvel at LLM performance in chemistry, physics, and math, we must ask: are these models *thinking* or *remembering*? Echoing Feynman's dictum about explaining things simply, an analogous test for AI might be: if you cannot answer a question you have never seen, you never truly understood.
The paper's value lies not only in exposing a specific problem (memorization in molecular property prediction) but in posing a universal question: how do we distinguish AI "memory" from AI "understanding"?