Paper Overview
- Field: Machine Learning
- Authors: Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron, Roland C. Aydin, Christian Feiler
- Published: 2026-09-04
- arXiv: 2609.05381
- Verbatim retrieval is widespread but benchmark-specific: on five datasets, more than 50% of the LLMs show verbatim retrieval, while on the remaining datasets it appears only in isolated cells.
- Reasoning changes retrieval behavior: the same experiments, on the same molecules and with the same prompt, are flagged 89% more often at the higher reasoning level than at the lowest one.
- Interruption attempts often fail: in the most contaminated cases, the strongest models can still identify combinations of transformed SMILES strings with original labels.
- Retrieval suppression narrows model differences: when retrieval is suppressed, prediction errors across different models converge in relative terms, whereas their differing use of verbatim retrieval disperses errors. This suggests that LLMs' general predictive ability is not determined solely by the amount of memorized values.
Summary
Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy alone cannot distinguish a model that genuinely predicts a property from one that simply retrieves a published number. This paper audits 22 frontier models on 12 regression benchmarks for verbatim retrieval of known values.
Key Findings
Conclusion
The work offers a systematic overview of the extent and depth of verbatim retrieval in LLM-based molecular regression benchmarks, highlighting the need to separate memorization from genuine predictive capability when evaluating models.
--- *Auto-collected on 2026-09-09*