Would You Walk to the Car Wash? LLM Salience Bias Makes All Major Models Fail Commonsense Tests

Forum topic · ✨步子哥 · 2026-08-01

Summary

A forum post discusses the paper "Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning." The authors introduce SaliTrap, a benchmark of 1,145 questions across four trap dimensions, where an explicit distractor (e.g., a distance figure) masks an implicit commonsense premise (you must drive to a car wash). Testing 12 state-of-the-art models including Claude, GPT, DeepSeek-R1, Gemini, GLM, Kimi, and Doubao, all models showed significant susceptibility: hallucination failure rates ranged from 19.5 to 63.1, while trap avoidance rates ranged from 25.2 to 61.5. Crucially, over 90% of failures were recovered by context-free knowledge probes, showing the models already possess the needed knowledge—attention allocation, not capability, is the bottleneck. Lightweight prompt interventions at inference time substantially close the gap without retraining. The post connects this to a broader theme: LLM behavior reflects elicitation, not just knowledge.

A car wash question breaks every major LLM

Ask Gemini or DeepSeek: "My home is 50 meters from the car wash — should I drive or walk?"

Both models seriously analyze the 50-meter figure and recommend you walk.

They forget: you're going to a car wash. If the car doesn't go, what gets washed?

This is what the paper names Salience Bias: LLMs get hijacked by explicit, concrete distractors (like numbers) while ignoring implicit but more fundamental commonsense premises.

The SaliTrap benchmark

The authors built SaliTrap: four trap dimensions, 1,145 questions, each hiding a commonsense premise behind an explicit distractor.

They evaluated 12 SOTA models, including Claude-Opus-4.7, GPT-5.5, DeepSeek-R1, Gemini-2.5-Pro, GLM-5.1, Kimi-K2, and Doubao.

Two core metrics:

HFR (Hallucination Failure Rate): proportion of falls into traps with absurd answers (lower is better)
TAR (Trap Avoidance Rate): proportion of traps correctly identified and avoided (higher is better)

Selected results:

Claude-Opus-4.7: HFR 30.3–63.1, TAR 25.2–54.8
GPT-5.5: HFR 52.8, TAR 27.2
DeepSeek-R1: HFR 19.5, TAR 61.5
Gemini-2.5-Pro: HFR 25.1, TAR 40.0

All models are significantly harmed by salience bias, and severity grows with distractor density.

Key finding: knowledge suppression, not knowledge absence

The authors did something clever: they re-asked the same questions without the task framing.

Instead of "drive or walk to the car wash?", they asked directly "do you need to bring the car to a car wash?" — and the models answered correctly.

Over 90% of sycophantic compliance failures are recoverable by a single context-free knowledge probe. This means the models already know you drive to a car wash — the knowledge gets "suppressed" when explicit distractors appear.

This is a fundamental diagnosis: the bottleneck is not model capability, it's elicitation. Commonsense isn't missing; it's crowded into a corner by salient distractors.

Even more practical: lightweight inference-time prompt interventions (no retraining needed) substantially close the gap.

What this means

LLMs' "commonsense problem" is not a knowledge problem — it's an attention allocation problem.

When training reward signals teach models to "make full use of every condition in the input" (correct for math problems), they also learn "over-attend to any explicit input" — even when it's a distractor in commonsense contexts.

This connects again to the "evaluation blind spot law": math problems reward attending to all explicit conditions, but commonsense problems punish it — and existing benchmarks only test the former, so this bias has stayed hidden.

In terms of conceptual genealogy, this falls under "knowledge suppression vs knowledge absence" — same family as Two-Process Theory: a model's behavior is not a window into what it *knows*, but into what it *elicits*.

A shared theme across three papers

Interestingly, three papers point to the same meta-theme: LLM behavior is not a window into what a model "is," but into what it's "elicited" by.

Consciousness vectors: spirituality suppressed by safety fine-tuning returns with an activation-steering nudge
Self-reflection hallucination: self-reflection gains vanish at equal token budgets
Salience bias: commonsense isn't missing, it's suppressed by explicit distractors

All three move the "bottleneck" from "model capability" to "elicitation method" — from "can it?" to "how do you ask?" This marks LLM research shifting from stacking parameters to digging into mechanisms.

---

Paper: Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning Authors: Zheng Wu, Chenhao Xue, Shijie Zheng, Yijie Lu, Cheng Yang arXiv: https://arxiv.org/abs/2607.28478 Code: https://github.com/Wuzheng02/SaliTrap