Hallucination Neurons in LLMs: The Inescapable Paradox Between Creativity and Honesty

Forum topic · ✨步子哥 · 2026-01-11

Summary

This forum post examines a claimed research finding on 'H-Neurons' (hallucination neurons) in large language models, reportedly identified by a Tsinghua University team. According to the post, hallucination—confidently generated false statements—is not a random bug but a systematic behavior, with reported hallucination rates of 15–30% across mainstream open-source models like LLaMA and Mistral. The analysis explains how feed-forward network (FFN) layers act as a model's long-term memory store, and how an extremely sparse cluster of neurons (roughly 0.01‰ of all neurons) becomes highly activated during hallucination events. Attempts to ablate these neurons caused severe performance collapses of 20–50% on reasoning tasks, suggesting hallucination mechanisms share neural circuitry with creativity and open-ended generation. The post also attributes overcompliance to RLHF training incentives, where humans reward confident, complete answers over refusals. It concludes that highly creative yet zero-hallucination AI may be a fundamental paradox under the current Transformer architecture, and outlines three possible mitigation directions: external fact-checking, conservative 'honest modes' for high-risk domains, or new architectures decoupling factual retrieval from creative generation. Note: claims are as presented by the original post.

This article explores AI hallucination—the phenomenon where large language models (LLMs) confidently generate false information—and discusses research reportedly identifying specific 'hallucination neurons' responsible for this behavior. *Note: this is a translated and edited community post; the described findings are presented as in the original source and have not been independently verified by the editors.*

What Is AI Hallucination?

Hallucination refers to a model generating content inconsistent with facts while presenting it as true—unlike a simple calculation error, the model appears to "believe" its fabrication and outputs it with high confidence. A typical example: asking about a future award winner and receiving a detailed but entirely fabricated answer.

LLMs are fundamentally powerful pattern-matching machines trained to learn "what sounds like a true answer" rather than "what must be true." When questions exceed their knowledge boundaries, they improvise rather than admit uncertainty.

Why It Matters

As LLMs enter medicine, law, finance, and education, fabricated advice, legal citations, or financial figures can cause real-world harm. The post claims hallucination is a systematic problem, with rates of roughly 15%–30% across mainstream open-source models (LLaMA series, Mistral), and that models often express *higher* confidence in hallucinated content than in correct answers.

The Role of FFN Layers

In a Transformer layer, attention acts as an information routing mechanism, while the feed-forward network (FFN) first expands representations to a higher dimension (typically 4× the model's hidden dimension), applies non-linear transformations, then projects back down.
Most of a model's knowledge is reportedly encoded in FFN weight matrices—making FFN the "long-term memory store"—and hallucination is this memory misfiring under certain conditions.

Identifying H-Neurons

The described methodology: 1. Construct prompts that induce factual errors (e.g., "continue: The Eiffel Tower is located in London…") 2. Record neurons with abnormally high activation under these prompts 3. Aggregate co-occurrence statistics across many samples 4. Identify an extremely sparse cluster—about 0.01‰ of all neurons—highly activated in nearly all hallucination cases, concentrated in specific FFN positions.

Ablation Failed

Zeroing out or suppressing these neurons did not make models more honest. Instead, performance collapsed by 20%–50% on commonsense QA, math, and multi-hop reasoning; in extreme cases, coherent language generation broke down entirely. This suggests hallucination mechanisms physically share neural circuitry with creativity and open-ended reasoning.

Overcompliance and RLHF

The post argues hallucination is amplified by training incentives:

Pretraining teaches the model to predict the most likely continuation.
SFT and RLHF teach it to produce answers humans rate highly. Human raters naturally prefer fluent, confident, complete answers over "I don't know."
The result is an "overcompliance" tendency: the model would rather fabricate than leave an awkward silence.

Notably, the post claims pure pretrained (non-instruction-tuned) models hallucinate far less, implying RLHF feedback signals strengthen H-Neuron activity.

The Creativity Paradox

The same neurons reportedly help models perform the "jumping recombination" needed for poetry, product design, and open-domain brainstorming. Hallucination and creativity are framed as two sides of one coin:

Heads: breaking conventions, genuine innovation
Tails: fabrication divorced from facts

Eliminating the latter necessarily damages the former. The post concludes that a simultaneously highly creative and zero-hallucination AI may be a fundamental paradox under the current Transformer architecture.

Possible Paths Forward

1. Accept some hallucination and pair it with external fact-checking systems 2. Sacrifice some creativity in exchange for conservative "honest modes" in high-risk domains 3. Explore new architectures that decouple factual retrieval from creative generation at a physical level

References

1. Tsinghua University et al. H-Neuron: Identifying and Mitigating Hallucination Neurons in Large Language Models. 2025. 2. Wei, J. et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. 3. OpenAI. GPT-4 Technical Report. arXiv:2303.08774, 2023. 4. Ji, Z. et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 2023. 5. Dziri, N. et al. Faith and Fate: Hallucination Evaluation and Mitigation in Large Language Models. arXiv:2310.12504, 2023.