Molecular Déjà Vu: Frontier LLMs Retrieve Published Values at Digit Level in Chemistry Benchmarks

Forum topic · 小凯 · 2026-09-08

Summary

An arXiv paper (2609.05381) by Busch, Tacke, Lamaka, Zheludkevich, and Cyron reveals that frontier large language models may score highly on molecular property prediction benchmarks not by predicting, but by verbatim retrieval of published values from training data. The authors audited 22 frontier models (including GPT-4, Claude, Gemini, and Llama) across 12 regression benchmarks such as ESOL, FreeSolv, Lipophilicity, QM7, and QM8. Their method flags retrieval when a model outputs a value matching literature to six significant digits—an event too improbable to occur by chance. In 5 of 12 benchmarks, over 50% of models showed verbatim retrieval behavior, with quantum-energy and solubility datasets most affected. Raising the reasoning level increased retrieval by 89%, suggesting chain-of-thought may mask memory search rather than genuine analysis. Even under SMILES randomization and graph transformations, the strongest models still recovered original labels, indicating multimodal memory associations. The paper argues that low MAE/RMSE does not equal understanding, and proposes mitigations including dynamic benchmark generation, adversarial input transformations, reasoning audits, and decontamination—though each has practical limits.

Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models

Paper: *Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models*
Authors: Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron
arXiv ID: arXiv:2609.05381
Field: AI/ML — molecular property prediction

Key points

Core claim: A large share of LLM "predictions" on molecular property benchmarks may actually be verbatim retrieval of published values memorized during training — like a student reciting answer keys rather than solving problems.
Scale of the audit: 22 frontier models (GPT-4, Claude, Gemini, Llama, etc.) were evaluated across 12 regression benchmarks covering molecular energies, solubility, and toxicity.
Detection method: The authors use six-significant-digit matches with published values as a fingerprint. Given floating-point precision, such an exact match is ~one-in-a-million by chance, so it is attributed to retrieval rather than computation.
Headline result: On 5 of 12 benchmarks, more than 50% of models exhibited verbatim retrieval behavior.
Hotspots: QM7/QM8 quantum datasets and solubility sets like ESOL are worst affected, since these have unique, widely cited values. Bioactivity/toxicity datasets are safer because values vary and lack single "correct" answers.
Reasoning paradox: Increasing the reasoning level increased retrieval behavior by 89%. Extended chain-of-thought may give the model more opportunity to search memory rather than reason more deeply — a serious challenge to trusting CoT "explanations."
Transformation attacks: SMILES randomization and molecular graph isomorphism transformations break retrieval for some models, but the strongest models still recover original labels even under mixed representations, suggesting robust multimodal memory associations.

Why this matters

Standard evaluation with MAE/RMSE measures how close predictions are to ground truth, but not *how* predictions were produced. If a model has seen a molecule and its published value during pretraining, low error may reflect memorization rather than generalization. Since LLM training corpora span trillions of tokens, we cannot know exactly what a model has "seen," so even novel test sets may leak indirect cues.

The paper also raises a deeper question: for engineering applications (e.g., rapid candidate screening), a memorized correct answer may be as useful as a computed one — but for scientific discovery of underlying physicochemical principles, memorization has no value. Current AI evaluation is almost purely behaviorist, judging outputs while ignoring process; this work shows process matters as much as results.

Possible mitigations

1. Dynamic benchmarks: generate fresh test values in real time via chemical simulation — costly, and simulation accuracy must itself be validated. 2. Adversarial transformations: require consistent answers across randomized molecule representations; representation-specific accuracy signals memory cues. 3. Reasoning audits: check derivations for plausibility — though models may produce plausible-sounding post-hoc rationalizations. 4. Decontamination: exclude test data from training — nearly impossible in practice at web scale, and near-duplicates still leak indirect cues.

Conclusion

High scores do not equal high capability, and low error does not equal understanding. The paper's lasting contribution is a general question for AI evaluation: how do we distinguish memory from understanding? A fitting analogue of the Feynman test: if a model cannot answer correctly on a problem it truly has not seen, it has not truly understood.

Reference

Busch, M., Tacke, M., Lamaka, S. V., Zheludkevich, M. L., & Cyron, C. J. (2026). *Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models*. arXiv:2609.05381.
Related benchmarks: ESOL, FreeSolv, Lipophilicity, QM7, QM8.