LLM Hallucinations Are Predictable: Model Size and Training Data Frequency Determine Factual Recall

Forum topic · 小凯 · 2026-05-19

Summary

A study by Smith, Shock, Segun, Olatunji, and Bissyandé shows that LLM hallucinations follow a predictable scaling law rather than occurring randomly. Factual recall quality is an S-shaped (sigmoid) function of a log-linear combination of model parameter count and the frequency of a topic in training data. Testing 38 models on over 8,900 academic citations via an automated citation-verification system, the researchers found these two variables explain 60% of recall variance across 16 dense models from 4 families, rising to 74–94% within a single model family. The theoretical explanation borrows the signal-to-noise ratio framing from the superposition hypothesis: recall signal scales with concept frequency while the noise floor is set by model capacity. A practical implication is that hallucinations on niche topics can be reliably predicted when training exposure is low. Open questions include whether the law holds for unstructured or contentious topics, multi-step reasoning tasks, and post-trained models.

Key Findings

LLM hallucinations appear random, but Smith, Shock, Segun, Olatunji, and Bissyandé show they follow a predictable scaling law: factual recall quality is a sigmoid function of a log-linear combination of model parameter count and the frequency of a topic in training data.

Two variables explain 60% of the variance in factual recall across 16 dense models from 4 model families.
Within a single model family, the explained variance rises to 74–94%.
Experiments covered 38 models tested on 8,900+ academic citations, using an automated citation-verification system to check whether models accurately recall reference details.
Larger models and more frequent papers in training data both lead to more accurate recall. These two factors dominate; other variables (architecture differences, training details) matter far less.

Theoretical Explanation

The explanation borrows the signal-to-noise ratio framing from the superposition hypothesis:

The "signal" strength of factual recall is proportional to how often a concept appears in training data.
The "noise floor" is determined by model capacity.
Larger models have lower noise, but whether a fact is recalled depends on the ratio of signal strength to noise level.

Practical Implication

If a niche topic appears only a few times in training data, you can fairly reliably predict that the model will hallucinate on related queries.

Open Questions

Academic citations are a relatively structured domain — do hallucinations on unstructured, contentious topics (political opinions, medical advice) follow the same law?
The study tests recall, not reasoning — do hallucination scaling laws differ on multi-step reasoning tasks?
Which 38 models were included — do they cover post-trained models (RLHF, DPO, etc.) across different training data distributions?

References

1. Smith, M. L., Shock, J. P., Segun, S. T., Olatunji, I. E., & Bissyandé, T. F. (2026). *Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency*. arXiv:2605.18732 [cs.CL]. 2. Kaplan, J., et al. (2020). *Scaling Laws for Neural Language Models*. arXiv. 3. Elhage, N., et al. (2022). *Toy Models of Superposition*. Transformer Circuits.