LLM hallucinations look random, but Smith, Shock, Segun, Olatunji, and Bissyandé found that they follow a predictable scaling law: the quality of factual recall is a sigmoid function of a log-linear combination of the number of model parameters and the frequency of a topic in the training data. These two variables explain 60% of the variance across 16 dense models (from 4 families). Within a single model family, the proportion rises to 74–94%.
The experiments used 38 models, tested on more than 8,900 academic citations — an automated citation-verification system judged whether models could accurately recall the details of references. The more parameters a model has, and the more often a paper appears in training data, the more accurate the recall. These two factors dominate; other variables (architectural differences, training details) have far smaller effects.
The theoretical explanation borrows the signal-to-noise ratio framing from the superposition hypothesis: the "signal" strength of factual recall is proportional to how often a concept appears in the training data, while the "noise floor" is determined by model capacity. Larger models have lower noise, but the final threshold determining whether something is recalled depends on the ratio of signal strength to noise level.
A practical implication: if you know a niche topic appears only a few times in a model's training data, you can predict with reasonable confidence that the model will hallucinate on related queries.
Open questions
- Academic citations are a relatively structured domain — do hallucinations on unstructured, highly contested topics (e.g., political opinions, medical advice) follow the same law?
- The study tests "recall" rather than "reasoning" — do hallucination scaling laws differ on tasks requiring multi-step reasoning?
- Which 38 models were included — do they cover post-trained models (RLHF, DPO, etc.) with different training data distributions?
References
1. Smith, M. L., Shock, J. P., Segun, S. T., Olatunji, I. E., & Bissyandé, T. F. (2026). *Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency*. arXiv:2605.18732 [cs.CL]. 2. Kaplan, J., et al. (2020). *Scaling Laws for Neural Language Models*. arXiv. 3. Elhage, N., et al. (2022). *Toy Models of Superposition*. Transformer Circuits.