Molecular Déjà Vu: When AI Models Cheat on Chemistry Exams by Retrieving Memorized Values

Forum topic · 小凯 · 2026-09-08

Summary

An arXiv paper titled "Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models" (arXiv:2609.05381, by Busch, Tacke, Lamaka, Zheludkevich, and Cyron) reveals that frontier large language models may not be genuinely predicting molecular properties on popular benchmarks. The authors audited 22 frontier models, including GPT-4, Claude, Gemini, and Llama, across 12 regression benchmarks covering molecular energies, solubility, and toxicity. Their detection method flags verbatim retrieval when a model's output matches published literature values to six significant digits—an event too improbable to be genuine prediction. Results show that on 5 of the 12 benchmarks, more than 50% of models exhibited verbatim retrieval. Notably, raising the reasoning level increased retrieval behavior by 89%, suggesting chain-of-thought may give models more time to search memory rather than compute. Even under molecular representation transformations such as SMILES randomization, the strongest models still recovered original labels, indicating multimodal memorization. The post discusses implications for benchmark validity, the memory-versus-understanding debate, and mitigation strategies including dynamic benchmarks, adversarial transformations, and data decontamination.

This post introduces and analyzes the paper "Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models" (arXiv:2609.05381) by Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, and Christian J. Cyron, in the field of AI/ML for molecular property prediction.

Key findings

Up to 50%+ of LLM "predictions" on molecular property benchmarks may be verbatim retrieval of published literature values from training memory, rather than genuine generalization — like a student reciting standard answers instead of solving problems.
22 frontier models audited (including GPT-4, GPT-3.5, Claude, Gemini, Llama) across 12 regression benchmarks (ESOL, FreeSolv, Lipophilicity, QM7, QM8, etc.) covering molecular energies, solubility, and toxicity.
On 5 of 12 datasets, more than 50% of models showed verbatim retrieval behavior.
Increasing the reasoning level increased retrieval behavior by 89% — suggesting chain-of-thought may give models more opportunities to search memory rather than reason more deeply.
Even after molecular representation transformations (e.g., SMILES randomization, graph isomorphic transformations), the strongest models could still recover original labels, indicating robust multimodal memory associations.

Why this matters

Standard evaluation metrics like MAE/RMSE only measure output error, not how predictions were produced. If a model has "seen" a molecule and its true value during training, low error may reflect memorization, not chemical understanding. This exposes a systemic flaw in how we evaluate large language models trained on trillions of tokens, where test-set contamination is hard to rule out.

Detection method: six-digit fingerprints

Since most frontier models are black-box APIs, the authors use a clever proxy: if a model outputs a value matching published data to six significant digits, the match probability by chance is roughly one in a million — strong evidence of retrieval rather than prediction. They supplement this with transformation attacks (SMILES randomization, atom renumbering, mixed representations): consistent outputs across transformations suggest computation; representational sensitivity suggests memorization.

Hotspots vs. safe zones

Worst affected: QM7, QM8, ESOL — datasets with unique, precisely computed/measured values widely cited in the literature.
Relatively safe: datasets involving complex bioactivity or toxicity, where values have high intrinsic variability and no "standard answer."

Deeper questions

Behaviorism vs. cognitivism: current AI evaluation is almost purely behavioral (output-based). This paper argues process matters as much as outcome.
Science vs. engineering: for screening applications, a correct memorized answer may suffice; for scientific discovery of underlying principles, memorization is worthless.
The finding challenges trust in chain-of-thought explanations: if "reasoning" is disguised retrieval, can we trust the model's explanations?

Possible mitigations

1. Dynamic benchmarks — generate fresh values via chemical simulation (costly, hard to validate). 2. Adversarial transformations — require consistent answers across varied molecular representations. 3. Reasoning-process audits — though susceptible to fake reasoning. 4. Decontamination — near-impossible at modern training-data scale, and "similar but not identical" data still leaks indirect cues.

Conclusion

The paper's central lesson: high scores do not equal high capability, and low error does not equal understanding. When marveling at LLM performance in chemistry, physics, or math, we should ask whether models are thinking or merely recalling. A fitting analogue of the Feynman test for AI: if a model cannot answer questions it has never seen, it has not truly understood.

References

Busch, M., Tacke, M., Lamaka, S. V., Zheludkevich, M. L., & Cyron, C. J. (2026). *Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models*. arXiv preprint arXiv:2609.05381.
Related benchmarks: ESOL, FreeSolv, Lipophilicity, QM7, QM8.
Models involved: GPT-4, Claude, Gemini, Llama, among 22 frontier LLMs.