Molecular Déjà Vu: Auditing Verbatim Retrieval of Published Values in Frontier LLMs on Molecular Property Benchmarks

Forum topic · 小凯 · 2026-09-09

Summary

Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy scores cannot distinguish genuine property prediction from verbatim retrieval of published values. A new arXiv paper (2609.05381) by Busch, Tacke, Lamaka, Zheludkevich, Cyron, Aydin, and Feiler audits 22 frontier models across 12 molecular regression benchmarks. Verbatim retrieval proves widespread but benchmark-specific: on five datasets, more than 50% of models show verbatim retrieval, while on the rest it appears only in isolated cases. Experiments at two reasoning levels show that reasoning changes retrieval behavior—identical molecules and prompts are flagged 89% more often at the higher reasoning level than at the lowest. The authors also test methods to interrupt retrieval in the most contaminated cases, finding that the strongest models can sometimes still recognize combinations of transformed SMILES strings with original labels. Notably, suppressing retrieval makes prediction errors across different models more similar, suggesting that general predictive ability is not determined solely by memorized values.

Overview

Field: Machine Learning Authors: Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron, Roland C. Aydin, Christian Feiler Published: 2026-09-04 arXiv: 2609.05381

Key Findings

Accuracy cannot distinguish prediction from retrieval. LLMs evaluated on molecular property benchmarks may simply be retrieving published numbers rather than predicting properties. This paper performs a verbatim retrieval audit to quantify the problem.
Widespread but benchmark-specific contamination. Across 22 frontier models and 12 regression benchmarks, verbatim retrieval is common: on five datasets, more than 50% of the LLMs show verbatim retrieval, while on the remaining datasets it appears only in isolated cells.
Reasoning changes retrieval behavior. Experiments run at two reasoning levels show that the same experiments—same molecules, same prompts—are flagged 89% more often at the higher reasoning level than at the lowest one.
Interrupting retrieval is hard. In the most contaminated cases, the authors test methods to interrupt retrieval. The strongest models can, in some cases, still recognize combinations of transformed SMILES strings with the original labels.
Retrieval spreads errors apart. Suppressing retrieval makes the prediction errors of different models closer in relative terms, whereas their differing use of verbatim retrieval disperses errors. This suggests that an LLM's general predictive capability is not determined solely by the amount of memorized numerical values.

Original Abstract (excerpt)

> Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy cannot distinguish a model that predicts a property from one that retrieves a published number. We audit 22 frontier models on 12 regression benchmarks for verbatim retrieval and find that it is widespread but relatively benchmark-specific: on five datasets more than $50\%$ of the LLMs show verbatim retrieval, while on the remaining datasets it appears only in isolated cells. We run our experiments at two reasoning levels and find that reasoning changes retrieval. The same experiments, on the same molecules and with the same prompt, are flagged $89\%$ more often at the higher reasoning level than at the lowest one. Finally, we test a way to interrupt retrieval in our most contaminated cas...

Paper: https://arxiv.org/abs/2609.05381