Don't Let Models Overthink: From Chain-of-Thought to Multimodal Reasoning — When Thinking Hurts

Forum topic · ✨步子哥 · 2026-01-08

Summary

A comprehensive review synthesizes two recent studies on when Chain-of-Thought (CoT) prompting helps or harms large language and multimodal models. The ICML paper 'Mind Your Step (by Step)' identifies six psychological task archetypes where verbal thinking degrades human performance, showing CoT causes significant drops on three of them: implicit statistical learning (GPT-4o falling from 87.5% to 64.4%), face recognition via verbal overshadowing (all six vision-language models declined), and classification with exception rules. The Wharton Generative AI Labs technical report, testing GPQA Diamond with 25 repetitions per question, finds CoT yields diminishing returns: reasoning models like o3-mini gain only ~3%, while CoT imposes a 20–600% latency tax and increases answer variance. The article builds a practical decision framework: avoid CoT for tasks relying on non-verbalizable statistical structure, fine-grained visual discrimination, or exception-laden rules, and for reasoning models where internal deliberation already exists. These findings are especially critical for vision-language models, where forced verbalization can divert attention from visual evidence.

Introduction: Longer Reasoning Is Not Always Better

Over the past two years, "Chain-of-Thought (CoT) / step-by-step thinking" has become a default incantation. But two core sources — the ICML-style paper *Mind Your Step (by Step)* and a Wharton Generative AI Labs technical report — point to a more mature conclusion: reasoning is an intervention, not a performance switch. It can significantly help some tasks and systematically hurt others, and the failures are common enough to matter.

Method Family 1: Explicit Verbal Reasoning (CoT / ToT)

CoT asks models to generate an intermediate token sequence ("think step-by-step") to raise the probability of a correct final answer — reasoning happens in language space.

When Does CoT Systematically Hurt?

*Mind Your Step (by Step)* builds an evaluation set from 6 task archetypes where more thinking makes humans worse. On 3 of them, CoT caused significant — sometimes cliff-like — drops across SOTA models.

#### 1. Implicit Statistical Learning

Using artificial grammar learning (finite-state grammars) with 4,400 classification problems, most models got worse with CoT:

GPT-4o: 87.5% → 64.4% (−23.1%)
Claude 3 Opus: 70.7% → 62.7% (−8.0%)
Gemini 1.5 Pro: 68.0% → 61.95% (−6.05%)
Llama 3.1 70B: 65.9% → 57.1% (−8.8%)

Tree-of-Thought only marginally helped (GPT-4o: 64.55% vs 62.52%), still far below the ~94% zero-shot subset score.

Takeaway: when tasks depend on statistical structure that resists verbalization, forcing verbal reasoning can push models onto wrong explicit rules.

#### 2. Verbal Overshadowing (Faces)

Adapting the classic verbal overshadowing effect to multimodal face recognition (5-way choice, candidates sharing identical text descriptions): all six vision-language models declined:

GPT-4o: 64.0% → 51.2% (−12.8%)
Gemini 1.5 Pro: 66.0% → 54.6% (−11.4%)
Claude 3 Opus: 44.0% → 29.6% (−14.4%)
Claude 3.5 Sonnet: 97.8% → 94.8% (−3.0%)

A simplified binary same-person task also dropped (GPT-4o: 72% → 62%), showing the interference is intrinsic to the reasoning process.

Takeaway: when discrimination relies on fine-grained visual differences that language describes coarsely, CoT is carving jade with a blunt tool.

#### 3. Rules with Exceptions

Reproducing the Williams et al. psychology paradigm (a feature "80% predictive with exceptions" plus a uniquely identifying feature like a license plate), CoT favored seemingly generalizable rules and slowed learning:

GPT-4o: average rounds 2.9 → 12.5 (+331%)
Claude 3.5 Sonnet: 2.3 → 6.4 (+178%)
Claude 3 Opus: 2.4 → 5.5 (+129%)

GPT-4o's CoT plateaued around 80% after 15 rounds — exactly the proportion of non-exception samples.

Takeaway: CoT's explanation preference can lock models into elegant but wrong simplifications.

Method Family 2: Psychology as an Evaluation Heuristic

The paper's higher-level contribution: tasks where human verbal thinking hurts serve as risk predictors for CoT. Bootstrap comparison against Sprague et al. (2025)'s 378 zero-shot vs CoT comparisons shows these 6 task types are significantly more extreme in both magnitude and frequency of drops (p < 0.00011 level). CoT failures are a task-structure-predictable systemic phenomenon, not random accidents.

Method Family 3: Industrial Evaluation — 25 Runs per Question

The Wharton report emphasizes evaluation rigor:

25 repetitions per model per prompting condition on GPQA Diamond (198 PhD-level science MCQs), since single-run evaluation masks instability.
Multiple threshold metrics: 100% correct, 90% correct, 51% correct — conclusions change with the threshold.

Evaluation Findings: Diminishing Returns and a "Time Tax"

Non-reasoning models

CoT raises average performance but increases variance, causing errors on questions the model would otherwise answer correctly.
Strong gains: Gemini 2.0 Flash (+13.5%), Sonnet 3.5 (+11.7%)
But on 100% correct rate: Gemini Pro 1.5 −17.2%
Cost: CoT requests are 35%–600% slower (about 5–15 seconds).

CoT sometimes pulls the model off an intuitively correct path onto a verbose but unstable one — averages can rise while the ceiling (perfect consistency) falls.

Reasoning models

Near-marginal gains: o3-mini (+2.9%), o4-mini (+3.1%); Gemini Flash 2.5 even −3.3%
Cost: still a 20%–80% time increase (10–20 seconds).

Core conclusion: when models already reason internally, adding generic CoT is redundant labor that occasionally interferes with built-in strategies.

A Research-Grade Map for "Should I Use CoT?"

High-risk task forms (Mind Your Step)

1. Implicit statistical structure (rules hard to verbalize) 2. Stimuli ill-suited to language representation (e.g., fine-grained face recognition) 3. Rules with exceptions plus more reliable memory/index features (CoT over-generalizes)

Cases where CoT is uneconomical even without score drops (Wharton)

1. Reasoning models that already "think" internally 2. Latency/cost-sensitive settings (20–80%+ time tax) 3. Stability/perfect-correctness matters more than average score

Why This Matters Especially for VLMs

The intuition that "CoT may be worse on VLMs" now has a rigorous anchor in the verbal overshadowing archetype: declines were observed for all tested LMMs on multimodal face recognition. When a task requires seeing more finely, CoT may force the model to say too much — diverting attention from visual evidence.

References

1. Liu, R., Geng, J., Wu, A. J., Sucholutsky, I., Lombrozo, T., & Griffiths, T. L. *Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse*. ICML. https://arxiv.org/html/2410.21333v4 2. Meincke, L., Mollick, E., Mollick, L., & Shapiro, D. *Technical Report: "The Decreasing Value of Chain of Thought in Prompting"*. Wharton Generative AI Labs, 2025-06-08. https://gail.wharton.upenn.edu/research-and-insights/tech-report-chain-of-thought/