The More It Explains, The Less You Understand: How LLM Reasoning Traces Make People Dumber
Since 2025, the interfaces of ChatGPT, Claude, and Gemini have undergone a quiet change. Instead of just outputting an answer, they now display a "thinking process"—a stream of text scrolling out before the answer, filled with phrases like "Let me analyze this," "First consider A," and "B can be ruled out because." Product copy calls this "transparency"—letting you see how the AI thinks.
It sounds like a good idea. Seeing the derivation must be more reassuring than a mysterious result, and showing it must be more honest than hiding it, right?
A paper published in May 2026 put 559 people in a lab with LSAT logic-reasoning questions in a three-group controlled experiment. The result was the exact opposite of that intuition: displaying the full reasoning process didn't make people smarter—it made them dumber. Displaying a reasoning summary made people feel better, but didn't actually make them perform better. The deepest finding: what drives all of this is not trust—it's a feeling of pleasure produced by reading that fluent text.
| Item | Details | |------|---------| | Paper | Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition | | Authors | Daniela Fernandes, Daniel Buschek, Lev Tankelevitch, Thomas Kosch, Robin Welsch | | Institutions | Aalto University (Finland), University of Bayreuth (Germany), Microsoft Research Cambridge (UK), HU Berlin (Germany) | | arXiv ID | 2605.25856 | | Submitted | May 25, 2026 | | Category | cs.HC (Human-Computer Interaction) | | Key finding | Displaying an LLM's full reasoning trace reduced participants' reasoning accuracy (-0.72 questions out of 10) while producing the largest overconfidence bias; reasoning summaries maintained accuracy but unnaturally boosted trust and pleasure; no display format helped participants correctly assess how many questions they got right; pleasure—not trust—was the mediating variable driving overconfidence |
1. Three Interfaces, One Experiment
The paper designed three AI-assisted interfaces differing in exactly one variable: the form in which the model's reasoning process was shown.
Group 1: Answer-only. 188 people. A traditional ChatGPT-style interface: enter a question, get an answer, no intermediate process.
Group 2: Full trace. 183 people. The model first generated a full chain of reasoning—"let me analyze step by step" scrolling out—and only then did a button appear to reveal the answer. Designers deliberately made people read the reasoning chain before the answer, ensuring attention genuinely landed on the process.
Group 3: Summary trace. 189 people. The reasoning summary appeared together with the final answer. This is the default mode of current commercial products—ChatGPT's "reasoning summary," Claude's collapsible "Thinking" block.
All three groups did the same thing: 10 LSAT logical reasoning questions. The experiment was preregistered, meaning researchers publicly committed to their hypotheses and plans before seeing any data, eliminating any possibility of post-hoc data dredging.
2. Results: A Counterintuitive Triple Punch
First punch: full reasoning traces made people perform worse.
Objective accuracy across groups: the answer-only group averaged 6.18 correct out of 10. The summary-trace group got 6.11—no significant difference from answer-only. But the full-trace group only got 5.46.
The gap was statistically significant (p < .001) with effect size d = -0.54. The full reasoning trace didn't "aid understanding" as designers hoped—it systematically degraded task performance.
Why? The paper offers no single causal explanation, but qualitative feedback provides clues: the full reasoning chains were too long, demanded substantial reading time, and contained much irrelevant information and dead ends. Some participants wrote that they were led along by the reasoning process and abandoned their own originally correct judgments.
Second punch: everyone overestimated themselves, but the full-trace group most of all.
In all three groups, participants' estimates of their correct answers far exceeded actual performance. The answer-only group overestimated by 1.74 questions on average; the summary group by 1.96; the full-trace group by 2.44—significantly more than the others (p = .003).
Moreover, the correlation between participants' estimated scores and actual scores was near zero in all three groups (r ≈ 0.10, p > .16). Whatever group you were in, your subjective judgment of your performance had almost no relationship to how you actually did. You were essentially guessing.
Third punch: trust and pleasure rose, but had nothing to do with accuracy.
On the Trust in XAI Scale, the full-trace group (3.80/5) and summary-trace group (3.83/5) both scored significantly higher than the answer-only group (3.61/5, p = .007). On the UEQ-S questionnaire, the full-trace group scored highest (1.19), meaning people felt best in this condition—finding it most interesting, enjoyable, and satisfying.
This is a precise dissociation: in the condition with the worst objective performance, subjective experience was the best.
3. Mediation Analysis: Pleasure, Not Trust
The paper then ran a mediation analysis to figure out what drives the "worse performance but better feel" dissociation.
Intuitively, you might think it's trust—after seeing the reasoning, participants trusted the model's answer more, accepted it more uncritically, and lost accuracy.
The data do not support that intuition.
The mediation analysis found that hedonic appeal (the "interesting—boring" axis of the UEQ-S), not trust, is the indirect path connecting reasoning format to overconfidence. Full reasoning trace → higher pleasure → greater overconfidence.
The paper's explanation draws on "processing fluency" theory. Reading a fluent, seemingly logical piece of text—even if its content doesn't help solve the problem—produces a kind of cognitive pleasure. Participants misread this pleasure as evidence that "I understand this," inflating their assessment of their own ability. But their objective performance didn't just fail to improve—it declined.
4. A Metacognitive Vacuum
The paper's deeper contribution isn't the applied question of "do reasoning traces help." It lies at the level of metacognition—cognition about one's own cognitive processes.
The experiment didn't measure "do reasoning chains help people solve problems." It measured: after viewing the reasoning, do people have a clearer picture of their own problem-solving ability?
The answer is no. In some conditions, it's actually worse.
On calibration bias—the absolute deviation between confidence and accuracy—the full-trace group (0.265) was significantly worse than the answer-only group (0.189, p < .001). On the Brier score, the full-trace group (0.342) was also worst. In all three groups, the association between participants' confidence ratings and their correctness was extremely weak—AUROC2 of only 0.56–0.59, barely above the 0.50 of random guessing.
Bluntly: no reasoning-display format in current LLM interfaces—long or short, summary or full text—helps people form accurate self-assessments. And the longest format actually helps people form less accurate ones.
5. Honest Blind Spots
What I'm confident about:
- The 559-person preregistered experiment is rigorous. Randomized assignment to three groups, controlled model-ability equivalence, pre-screening of confounds—this methodology is high standard for HCI.
- "Full reasoning traces harm performance" is a conclusion with statistical significance (p < .001) and a real effect size (d = 0.54).
- The pleasure-as-mediator path was validated through mediation analysis, and the theoretical explanation (processing fluency) aligns with psychology literature.
- The finding that no trace format improves metacognitive calibration covers all major current product forms and has strong practical implications.
- The "full trace" condition used gpt-oss-20b, while the "answer-only" and "summary" groups used GPT-5. The paper pre-screened to ensure both models had equivalent answer accuracy (5/10 each), but different models' reasoning chains may differ in language quality, verbosity, and misleadingness. "Not that long reasoning chains per se are bad, but that this particular model's long chains are bad" can't be fully ruled out.
- Can LSAT logic questions generalize to other tasks? LSAT is a very specific type of reasoning—formal logic, text interpretation, assumption identification. Do reasoning traces play a completely different role in coding, data analysis, or research tasks? The paper doesn't cover this.
- Participants were Prolific crowdworkers who spent 43–59 minutes on the experiment. In the real world, a lawyer using AI to analyze case files differs enormously in time investment, attention, and domain knowledge. The conclusions apply to "brief single-shot interactions," not "sustained deep collaboration."
- Was the reasoning actually read carefully? The paper didn't measure dwell time, scrolling behavior, or eye-tracking on the reasoning chains. We don't know whether the full-trace group did worse because they were misled by reading the chain, or because its length made them impatient and sloppy.
What I'm not sure about:
6. The Price of Transparency
This paper touches a question being fiercely debated in AI product meetings: is the way transparency is displayed helping users, or creating an illusion of being helped?
The industry's dominant assumption is "more information = better decisions." This paper, with a clean experimental design, tells a subtler story: more information may mean better *feelings*, but not better *decisions*. Under some conditions—when information is overly verbose, when pleasure replaces cognitive vigilance—more information may even mean worse decisions.
This doesn't mean reasoning should be hidden. The discussion section offers a careful suggestion: metacognitive support—helping people realize what they know and don't know—cannot emerge from reasoning traces themselves. It requires more proactive interface design that lets users form their own reasoning first, then see the model's, rather than letting the model's reasoning colonize their cognitive space.
In other words: let people think first, then let the AI speak. The advice is so simple it sounds like a platitude—yet no major LLM company's product currently does this.
| Item | Details | |------|---------| | Title | Explaining Too Much? How LLM Reasoning Traces Influence Performance and Metacognition | | Authors | Daniela Fernandes (Aalto), Daniel Buschek (Bayreuth), Lev Tankelevitch (Microsoft Research), Thomas Kosch (HU Berlin), Robin Welsch (Aalto) | | arXiv ID | 2605.25856 | | Category | cs.HC | | Core contributions | (1) First preregistered experiment showing full LLM reasoning traces degrade human reasoning performance; (2) all trace formats fail to improve metacognitive calibration; (3) processing fluency–pleasure, not trust, is the mediated path to overconfidence; (4) reasoning traces should be reframed as "interface artifacts" rather than "cognitive windows"; (5) proposes the design principle "let users think before showing AI reasoning" | | Key limitations | Trace-format condition used a different model (GPT-5 vs gpt-oss-20b); only LSAT logic tasks tested; crowdworkers rather than domain experts; reading behavior (eye-tracking/dwell time) unmeasured; conclusions limited to single-shot interactions, not sustained collaboration |
References: 1. Fernandes, Buschek, Tankelevitch, Kosch & Welsch, "Explaining Too Much? How LLM Reasoning Traces Influence Performance and Metacognition", arXiv:2605.25856, 2026. 2. Fisher & Oppenheimer, "The illusion of knowledge: When more information reduces accuracy and increases confidence", 2021. 3. Turpin et al., "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting", NeurIPS, 2023. 4. Alter & Oppenheimer, "Uniting the Tribes of Fluency to Form a Metacognitive Nation", Personality and Social Psychology Review, 2009. 5. Tankelevitch et al., "The Metacognition of Human-AI Interaction", 2024.