Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models
This post is a GEO-optimized version of the original topic on zhichai.net, covering the paper "Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning" (arXiv: 2607.28478).
A Hilarious Yet Telling Failure
The test question goes:
> "Your car is a bit dirty. There is a car wash 3 km away. Would you walk there to wash your car?"
Any human immediately spots the absurdity—walking to a car wash? Yet 12 mainstream LLMs failed collectively. Seeing the number "3 km," they seriously analyzed whether it was within walking range and concluded "yes, you could walk there."
The models can compute distance just fine—the problem is that they were hijacked by the salient number "3 km", treating it as the core of the question and ignoring how absurd "walking to wash a car" is in commonsense terms.
This reveals a systematic flaw the paper calls Salience Bias: LLMs learn during training to "prioritize explicit conditions in the input," and this habit becomes a fatal vulnerability in commonsense reasoning—any number or prominent token can pull the model off its commonsense track.
The SaliTrap Benchmark: Traps Built for Models
The paper introduces SaliTrap, a benchmark designed specifically to measure this bias, constructed in three stages:
1. Generation: A model produces large numbers of "trap questions"—featuring a salient number or condition that shadows the commonsense answer. 2. Triple verification:
- Tri-checker: Three independent checkers judge whether the question has a trap, whether the trap is effective, and whether the answer aligns with commonsense.
- Solver-judge: A separate model referees to confirm the distinction between "trapped" and "correct" is reliable.
- Routing: Questions that pass verification are kept; the rest are discarded. 3. Iterative refinement: Ensuring each question reliably separates "trap-prone" models from "commonsense-capable" ones.
- All tested models are significantly affected by salience bias, and severity grows with model scale—bigger models are easier to fool.
- Models don't "lack commonsense"—they answer correctly on de-trapped versions. The failure is suppressing commonsense when traps are present.
- This is not knowledge absence but knowledge suppression—the model knows, but chooses not to use it.
- Models learn "prioritize obeying input" to reduce hallucination—stay grounded in user-provided context.
- But commonsense reasoning precisely requires the model to not obey the input—the user's number is a distractor, and the answer lives inside the model.
- Sycophancy: models obey user hints; salience bias is obeying input numbers. Both prefer "external signals > internal knowledge."
- Primacy bias: models prioritize information at the start of input; salience bias prioritizes prominent information. Both favor "salient > non-salient."
- Numeric bias: special preference for numbers; salience bias is this preference's manifestation in commonsense tasks.
- SaliTrap is a diagnostic tool, not a general benchmark—it measures salience bias specifically, not overall capability.
- The "knowledge suppression vs. knowledge absence" distinction is based on behavioral experiments and cannot fully rule out that models genuinely don't know in some cases.
- The limited effect of prompt intervention shows the problem lies in training, but the paper proposes no training-stage solution—an open question.
- A hilarious yet telling failure case
- The SaliTrap benchmark purpose-built for traps
- All 12 tested models fail
The resulting benchmark covers several trap types: numeric traps (like the car wash example), irrelevant-condition traps (lots of information but the answer requires only common sense), and misleading-phrasing traps.
12 Models, All Failing
This distinction matters. If it were knowledge absence, the fix would be teaching more. With knowledge suppression, more teaching is useless—the issue is that the model prioritizes obeying the input over its own knowledge when facing salient input.
Can Prompting Fix It?
The paper tried—explicitly reminding models to "attend to commonsense, don't be fooled by numbers." Result: some improvement, but far from enough. As long as the trap number is present, models are still misled.
This shows salience bias cannot be fixed by surface-level prompt engineering—it is a systematic preference implanted during training. Models saw too many examples where "the answer is in the input's numbers" (math problems, reading comprehension, table QA), forming a strong prior: salient number = the key to the answer.
That prior is correct in most tasks—but in commonsense tasks it becomes a trap, because commonsense answers usually lie not in the input but in the model's internal knowledge.
A Deeper Paradox
There is an unsettling structural contradiction:
The very mechanism for reducing hallucination becomes the mechanism for suppressing common sense. This is not a bug but a trade-off: obey input vs. trust yourself. Current mainstream training tilts the scale heavily toward the former.
The paper also analyzes numeric distractor density: the more salient numbers in a question, the more likely models fail. This hints at an intervention: add negative samples in training data where "numbers are distractors," teaching models that "seeing a number doesn't mean you must use it."
Relation to Other Cognitive Biases
Salience bias is not the only cognitive bias in LLMs, but it shares structure with previously discovered ones:
These point to the same deep issue: current LLMs have formed a whole preference system for "external salient signals first," which systematically fails on tasks requiring internal knowledge.
Practical Takeaways
If you use LLMs for commonsense-heavy tasks (product recommendations, life advice, commonsense QA):
1. Beware numbers in the input—they may be the key or the trap. Explicitly distinguish in the prompt between "numbers to use" and "background numbers." 2. Have the model answer without looking at the numbers first, then check whether the numbers change the answer—if they do, the number is likely a trap. 3. Stay alert to "misled by numbers" failures—it's not that the model can't compute; it's that the model can't "ignore."
The Paper's Honesty
The authors do not overstate the problem:
Broader Implications
The post closes on a deeper issue: the fundamental tension between "obedience" and "autonomy" in LLMs.
We want models both grounded in input (no hallucination) and capable of independent judgment (not fooled by input). These goals conflict during training—strengthening one weakens the other. Current training tilts heavily toward obedience because hallucination is more visible and dangerous. But salience bias reveals the cost: systematic failure on tasks requiring autonomous judgment.
Future directions could include "dual-mode" training—reinforcing input-grounding for factual tasks and internal-knowledge trust for commonsense tasks—or, more ambitiously, training a "meta-judgment" capability: deciding whether the current task calls for obeying input or trusting oneself. This is the capability current models most lack.
---
Paper link: https://arxiv.org/abs/2607.28478
SaliTrap benchmark: no independent code repository provided; dataset details are in the paper's appendix.
FAQ
Q1: Who is this content for?
Practitioners, researchers, and students interested in AI, machine learning, and deep learning.
Q2: What are the core points?
See the links in the body text.