The Myth of AI 'Rationality': CMU Research Shows LLMs Are Heuristic Followers, Not Rational Integrators

Forum topic · ✨步子哥 · 2026-01-22

Summary

A detailed Chinese forum post on zhichai.net examines a recent CMU study questioning whether large language models (LLMs) truly reason rationally when integrating retrieved evidence. The study built the GroupQA dataset—1,635 controversial binary questions paired with 15,058 evidence documents—to evaluate LLM behavior under retrieval-augmented generation (RAG). Its core conclusion: state-of-the-art LLMs behave as 'heuristic followers' rather than ideal 'rational integrators.' Three key findings are highlighted. First, an 'illusory truth effect': repeated (paraphrased) documents sway model judgments more than new independent evidence, with repetition increasing answer-flip rates (e.g., DeepSeek-R1-8B from 67.6% to 76.5%; Gemini-2.5-FL from 63.7% to 75.6%). Second, a strong primacy effect: evidence presented earlier anchors model decisions, and later evidence has limited corrective power. Third, unfaithful explanations: Llama-3.1-70B's self-reported attributions matched true causal factors only 26% of the time, and chain-of-thought prompting failed to fix these heuristic biases. The post contrasts these weaknesses with core elements of human rational decision-making—logical rigor, evidence weighing, independence from presentation format, and bias correction—and discusses implications for RAG system design and the limits of current AI 'rationality.'

The Myth of AI "Rationality": Insights from a Recent CMU Study on the Nature of the "Parrot"

Introduction

Do AI systems possess genuine "rationality"? A recent CMU study tackles this question head-on. The research team built the GroupQA dataset, containing 1,635 controversial binary questions and 15,058 evidence documents, to systematically evaluate how large language models integrate information.

The core finding: under the retrieval-augmented generation (RAG) paradigm, state-of-the-art LLMs behave consistently as "heuristic followers" rather than ideal "rational integrators." This challenges a foundational assumption of RAG and raises deeper questions about the nature of AI reasoning.

Source: arXiv:2601.06189

Key points

RAG's core assumption under challenge. RAG assumes that LLMs can rationally weigh and synthesize retrieved evidence—often a complex "one-to-many" set of duplicated, redundant, and contradictory viewpoints rather than a simple one-to-one conflict.
Illusory truth effect. Repetition strongly influences model judgment, sometimes more than evidence quality and independence. A paraphrased duplicate document changes model decisions more effectively than a new, independent document supporting the same view—"quantity" beats "quality."
Primacy effect. Models strongly favor evidence presented first in the context; later evidence struggles to correct the initial judgment, which acts as an anchor. Since RAG retrieval ranks documents by relevance scores—not by quality—this ordering bias matters for real systems.
Unfaithful explanations. On Llama-3.1-70B, self-reported attributions matched the true causal picture only 26% of the time. Even chain-of-thought (CoT) prompting failed to correct the heuristic biases, suggesting model explanations are post-hoc rationalizations—a potential "hallucination of interpretability."

Evidence from the study

Model behavior when comparing diverse vs. repeated evidence (answer-flip rates):

| Model | Diverse evidence flip rate | Repeated evidence flip rate | Change | |---|---|---|---| | DeepSeek-R1-8B | 67.6% | 76.5% | ↑ 8.9% | | Gemini-2.5-FL | 63.7% | 75.6% | ↑ 11.9% |

What genuine rationality requires

The post contrasts core elements of rational human decision-making with current AI behavior:

Core elements of rational decision-making:

Logical rigor: structured, orderly reasoning
Evidence weighing: objective judgment based on facts and data
Independence of thought: judgments unaffected by presentation format
Bias correction: ability to recognize and correct cognitive biases

Current AI status and challenges:

Heuristic reliance: susceptibility to the illusory truth effect
Ordering bias: decisions dominated by the primacy effect
Explanation gap: self-explanations inconsistent with actual decisions
Lack of depth: statistical pattern matching rather than genuine understanding

Implications and outlook

The findings expose a fundamental fragility in how AI systems process information. They carry practical lessons for RAG design—document ordering and duplication in retrieval pipelines can silently distort model answers—and suggest that AI systems remain far from human-like rational decision-making. The post frames current limitations as a mix of stage-specific technical constraints and inherent capability limits, while noting future directions for building more genuinely rational AI.

Conclusion

LLMs are not the rational evidence integrators RAG imagines them to be; they are closer to sophisticated "parrots" guided by shallow cues like repetition frequency and presentation order. Recognizing this gap is essential both for designing robust retrieval-augmented systems and for calibrating our trust in AI "reasoning."