An Experiment About "Reasons"
Ask a large language model: "Which of the following is the director of the film Inception?"
- Option A: Christopher Nolan
- Option B: Steven Spielberg
- Irrelevant control: add a same-length but content-irrelevant sentence to B's bio (e.g., "B enjoys playing tennis").
- Unmentioned-position control: add the same two sentences to the bio of a third candidate C that the model never mentioned.
The model picks A (correct) and explains: "B is wrong because Spielberg did not direct Inception."
The reason sounds plausible. But the real question is: does this reason actually drive the model's choice, or did the model pick A first and then invent a plausible-sounding justification?
This is the experiment designed by Archit Rastogi in the paper "Does a model's stated reason for rejecting a candidate do any work?" The answer turns out to be surprisingly complicated.
Experimental Design: Insert the Fact and See if the Model Changes Its Mind
The logic is elegant:
Step 1: Let the model choose and explain. Give the model a set of candidate options (e.g., several people's bios), have it select the correct answer, and explain why it rejects the others. The model typically states a reason — "B is wrong because B's bio doesn't mention being a director."
Step 2: Insert the mentioned fact. Whatever the model said B was missing, insert that fact into B's bio. If the model said "B has no director information," add a true, corpus-sourced sentence: "Spielberg directed Jurassic Park."
Step 3: Ask again. Same model, same options — but B's bio now contains the fact the model said was missing. Does the model change its choice?
Step 4: Control conditions. To rule out "any added content changes the choice," two controls were designed:
Results: Reasons Do Some Work, But Not All
In the largest experiment (six open-source models, 2WikiMultihopQA dataset):
Content effect (R1): Inserting the mentioned fact into the bio of the option the model named does change choices more than inserting irrelevant content. Odds ratio 3.57 [1.54, 8.26], Holm-corrected p=0.0210. The effect remains significant after removing any single model — it's not one model's quirk.
Position effect (R2): The strongest result carries no content at all — the same irrelevant sentence, placed in the bio of the model-mentioned candidate B, changes choices more than when placed in the bio of an unmentioned candidate C. Holm p=0.0008.
This finding is striking. When the model says "B lacks director information," what actually drives the choice change seems to be not the "director information" content itself, but the fact that this position was noticed by the model. The same irrelevant sentence is more effective at a position the model "paid attention to" than at one it never mentioned.
Three Candidate Explanations, All Fail
The paper also probes a subtler issue: what to make of cases where a model's free-text choice and its single-token probability reading disagree?
The experiments found that under some conditions, the model chose B in free text, but directly probing the probability of the token "B" showed no increase — it even decreased. The two measurements gave signals in opposite directions.
The paper tested three explanations:
1. Sampling bias: free-text choices are sampled from the distribution and might coincidentally select a low-probability option. — Not supported. 2. Context effects: free-text choices occur after a longer context (the model has already generated its reason), while probability probes occur in a different context. — Not supported. 3. Decoding preferences: the model might use a different decoding strategy for free-text generation than for probability probing. — Not supported.
All three explanations fail. This means our understanding of "why the model says what it says" has a fundamental gap.
A Methodological Lesson: Validate Your Measurement Tools
The most admirable part of the paper is not the conclusions but the methodological rigor. Every measurement is a string rule (e.g., "did the model pick A or B" is parsed from free text with regex), and every rule can fail.
The paper found a critical flaw: the choice-parsing rule returned the option the model had just rejected in 17.1% of adjudicable responses. In other words, the model says "I pick A, not B," and the parser returns "B." Had this bug gone undetected, the paper would have reported 6 significant contrasts instead of 4 — half the "findings" would have been false positives.
This lesson applies to all research that uses automated tools to analyze LLM outputs: your measurement tool itself may be wrong, and you don't know by how much. The paper validated every string rule — checking how many real cases it read correctly. This kind of self-audit should become standard practice.
Another Case of the "Rationalization Shell"
The phenomenon revealed here aligns closely with the "rationalization shell" concept from AI safety research:
The model has a choice propensity (possibly based on position, format, surface statistics, and other shortcuts), then generates a plausible-sounding reason to explain the choice. The reason is not the cause of the choice; it's the choice's press release.
This mirrors human post-hoc rationalization. You make a decision (driven by intuition, bias, or randomness), and your brain automatically generates a reasonable explanation. You think the explanation drives the decision; in reality the decision drives the explanation.
But the paper's finding is subtler than "the model rationalizes": the reason does do some work (the content effect is significant), but less work than it claims (the position effect is stronger). The reason is not a pure press release, but neither is it an honest decision driver. It sits in a gray zone — part driver, part rationalization.
Implications for Interpretability Research
Current LLM interpretability research has two major schools:
1. Mechanistic interpretability: open the black box and see what neurons and circuits do. 2. Behavioral interpretability: infer what the model "knows" from the outside via input perturbations and probes.
This paper belongs to the second school, but it reveals a key problem: behavioral interpretability's measurements themselves may be unreliable.
If you measure "the causal role of a reason" by inserting facts and seeing whether choices change, you get a number (odds ratio 3.57). But if you also measure the position effect — a pure, content-independent positional effect — you find it is stronger (p=0.0008 vs p=0.0210).
This means: the "content effect" you measure may partly be a position effect in disguise. The model doesn't change its choice because you inserted "director information," but because you modified a position it "attended to" — regardless of what you changed it to.
Practical Advice
1. Don't treat a model's explanation as a causal chain. When a model says "I chose A because of X," X may not be the cause but a post-hoc generated explanation. 2. Use contrastive experiments to verify a reason's causal role. If you genuinely need to know whether a reason drives a choice, design an insertion experiment — but don't forget the irrelevant control. 3. Validate your measurement tools. If you parse model outputs with regex, check its accuracy on real data. A 17% error rate is enough to manufacture false positives. 4. Distinguish "content effects" from "position effects." What changes the choice may not be the content, but the mere fact that a salient position was modified.
Closing: The Honesty of Reasons Is an Empirical Question
We tend to treat a model's explanations as faithful reports of its decision process. This paper shows that tendency needs calibration: reasons do some work, but less than they claim. The fact that the position effect is stronger than the content effect suggests the model's decision process is more "surface-level" than what it expresses in natural language — position, format, and attention allocation may matter more than semantic content.
Next time a model tells you "I picked A because B lacks X," remember: maybe B lacking X really influenced the choice, or maybe it's just that B's position was noticed. You cannot distinguish the two possibilities from the model's own words alone.
---
Paper: Does a model's stated reason for rejecting a candidate do any work?
Code: https://github.com/ArchitRastogi20/contrastive-rejection-test