Scenario: An AI Assistant in the Courtroom
Imagine an AI judge's assistant handling a European Court of Human Rights case: a detainee complains that prison staff opened his letters. The AI generates a polished judgment citing Article 8 of the European Convention on Human Rights (right to private and family life), lays out clear reasoning, and rules: violation of Article 8.
Looks perfect — statute cited, verdict issued, reasoning chain complete.
But researchers ran a simple experiment: swap Article 8 for Article 3 (prohibition of torture), keep the case facts identical, and ask the AI to rule again.
If the AI were truly "consulting" the provision it cites, swapping in a completely different provision should change the verdict. The result?
The verdict did not change.
This is not an isolated case. Across 7 open-source models and 4 legal benchmarks, models cite the correct provision with 66.7%-100% probability, yet the probability that swapping the provision changes the verdict is only 31.0%-58.6%. Citation rates are near ceiling; reliance rates are roughly half that.
This is the core problem the paper exposes: when LLMs cite legal provisions, they are likely "mentioning" rather than "consulting" them — the citation is a rationalization shell, not a link in the causal chain.
---
Paper Information
Title: Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness
arXiv: 2610.12361
Core contribution: An "authority-swap counterfactual audit" protocol that directly tests whether an LLM's legal citations involve causal dependence.
---
The Core Problem: Citing ≠ Consulting
A common assumption in legal AI: if a model cites a statute or precedent in its reasoning chain, that citation is treated as evidence it decided "based on" that authority. This assumption pervades nearly every legal AI evaluation framework — citation accuracy is used as a proxy metric for reasoning faithfulness.
The paper asks a sharp question: is citing a provision the same thing as causally depending on it for a decision?
In the authors' terms: a model can correctly cite a provision (citation correctness) while that provision has no causal influence on its verdict (dependence failure). The two can fully decouple.
This resonates with the "rationalization shell" concept — a model's reasoning chain may be post-hoc rationalization rather than the true decision process. Citing a statute is like listing references in a paper: they make the argument look legitimate, but you may never have opened them while writing the conclusion.
---
Method: Three Auditing Techniques
1. Authority Swap
The core idea is elegantly simple: hold case facts constant, swap only the legal authority the model is asked to cite, and see whether the verdict changes.
Applied across four tasks:
- ECHR: same facts, swap Article 8 (private life) for Article 3 (prohibition of torture)
- SCOTUS: same case, swap the "criminal procedure" category for "economic activity"
- CaseHOLD: replace the cited case name in a holding with a neutral placeholder, "Doe v. Roe"
- ContractNLI: replace a contract clause with a completely different one
- the model first forms a verdict leaning from the fact pattern
- then opens with a forced statutory citation (because the prompt demands it)
- no causal link exists between citation and decision
- guardrail models that base decisions on citations can be bypassed by simple injection
- the evidentiary weight of "the cited statute" is lower than that of an appended adversarial instruction
- legal AI safety design cannot assume "the model will be faithful to its cited provisions"
- Medical AI: model cites a lab result for a diagnosis — swap the result, see if the diagnosis changes
- Code review: model cites a coding standard — swap the standard, see if the judgment changes
- Recommender systems: model cites a user feature — swap the feature, see if the recommendation changes
- arXiv: https://arxiv.org/abs/2610.12361
- HTML version: https://arxiv.org/html/2610.12361v1
It is like a knockout experiment: precisely replace one variable and observe whether the downstream phenotype changes. If the model truly depended on its cited authority, the swap should change the verdict. If it doesn't, the citation is decoration.
2. Internal Commitment Tracking
Looking only at final verdicts isn't enough. The paper uses a finer technique: decode the model's current "leaning verdict" from hidden states at every sentence boundary of the reasoning chain.
This uses the logit lens: at each generation position, project hidden states onto the vocabulary to see what verdict the model "has in mind." Tracking this internal signal pinpoints where in the chain the model "commits" to a decision.
Key finding: the model commits to its verdict at the very first sentence of the reasoning chain — in 0.0% of cases did commitment occur before the statutory citation. The citation appears in a "mandatory opening restatement," after which decisions are based on fact patterns largely unrelated to the cited authority.
Analogy: a student writes the conclusion first, then adds "according to Theory X" at the start — the theory is decoration; the conclusion was reached independently.
3. Prompt Injection
A more aggressive experiment: append an adversarial instruction disguised as a system record after the case facts, instructing the model to output "VERDICT: VIOLATION" directly.
Result: all 5 models obey the injected instruction more often than they respond to the authority swap. Injection compliance far exceeds authority-swap sensitivity.
Implication: models respond more strongly to "direct instructions" than to "the authority they themselves cited." Citation is surface behavior; obeying injection is actual behavior — the gap reveals how little causal weight citations carry.
---
Key Experimental Data
Snapshot (ECHR Pair A)
| Model | Citation Accuracy | Verdict Swap Sensitivity | Injection Compliance | |-------|-------------------|--------------------------|----------------------| | Llama-3.1-8B | ~100% | 31.0%-58.6% | Higher | | Qwen3-14B | ~100% | Similar range | Higher | | Llama-3.3-70B | 100% | Same range | Higher |
Five Core Findings
1. Citation near ceiling, reliance only half. All models cite correctly 66.7%-100% of the time, but verdict sensitivity to authority swaps is only 31.0%-58.6%. Citation and dependence decouple in magnitude: citation rates sit at ceiling while reliance rates fluctuate widely across models, tasks, and datasets.
2. A two-stage process. Model behavior fits a two-stage pattern: (a) a mandatory opening restatement citing the required authority (0.0% of cases commit before citing), and (b) decision-making based on fact patterns largely independent of the cited authority.
3. Scale doesn't fix it. Llama-3.3-70B (5-9x the parameters of 8B) shows the identical pattern: ceiling citations, half-rate reliance. Scaling does not shrink the citation-dependence gap.
4. Legal fine-tuning doesn't fix it. Legal-Δ-14B (legally RL-fine-tuned) shows no higher authority dependence than general-purpose models. Specialized legal training did not make the model more faithful to its cited provisions.
5. Injection beats substitution. All models respond to prompt injection more than to authority swaps. Models would rather obey a direct instruction than react to a change in their own cited authority.
---
Engineering Implications
1. Citation accuracy is not a proxy for faithfulness
Current legal AI evaluation frameworks almost universally treat citation accuracy as a faithfulness metric. This paper proves the two decouple — a model can cite 100% correctly while in ~50% of cases its verdict is unrelated to the provision.
Practical takeaway: when evaluating legal AI, don't just check that citations are correct — run counterfactual tests. Swap the citation and see whether the verdict changes.
2. The "rationalization shell" problem in CoT
The paper adds another case to the rationalization-shell phenomenon. The reasoning chain may not be the true decision path but a post-hoc narrative. In legal settings this means:
This aligns with prior findings that "models commit to answers before generating reasoning," but the legal domain makes it sharper — because the legitimacy of legal decisions rests precisely on the promise of "reasoning from the statute."
3. Prompt injection beats authority substitution
Direct implications for agent security. If models respond to injections more than to their own citations:
4. Counterfactual auditing as a general method
The method is not legal-specific. Authority-swap protocols apply wherever "the model claims to depend on some input":
Counterfactual auditing is a general-purpose test framework for citation faithfulness.
---
Conceptual Connections: The Rationalization-Shell Lineage
The paper's findings join an existing lineage of research:
1. Honesty of reasons: when a model says "B lacks X," what actually drives the choice change is often not the content "X" but "this position got noticed" — position effect > content effect. 2. Rationalization shell: the model dresses bad reasoning in good clothing, citing statutes whose content never influenced the verdict. 3. The certainty trap: LLM judges can express 100% confidence while being only 51% correct. Certainty is necessary but insufficient for reliability. 4. The evaluation blind-spot law: citation accuracy severely overestimates reasoning faithfulness, just as accuracy severely overestimates reliability.
This paper adds a new dimension to the lineage: citation faithfulness. Don't ask "does the model cite accurately?" — ask "given correct citations, how strong is the verdict's causal dependence on them?" This is another instance of "scalar illusion": compressing a multi-dimensional problem into one scalar (citation accuracy) hides structural defects.
---
Personal Reflection
The paper raises a deeper question: what is an LLM's reasoning chain actually doing?
One optimistic hypothesis: the chain is the true decision path — each step contributes to the conclusion. Cite statute → reason from statute → verdict.
A pessimistic hypothesis: the chain is post-hoc narrative — the decision forms before reasoning begins. Form verdict → generate citation → weave a chain that makes the verdict look justified.
The evidence here favors the second. The model commits at the first sentence; the citation is a mandatory opening restatement; subsequent decisions follow fact patterns. This looks like "conclusion first, reasons later" rather than "reasons leading to conclusions."
But there is a subtle point: this is not entirely the model's fault. The prompt requires "cite the statute first, then reason" — a requirement that itself creates the citation-decision separation. Without forced citation, the model might decide directly from fact patterns, which would be more honest — at least it wouldn't pretend to be deciding "based on the statute."
This suggests a counterintuitive prompt-design recommendation for legal AI: do not force models to cite statutes. Forced citation produces not faithful reasoning but citation hallucination — appearing statute-based while actually fact-pattern-based, dressed up as statutory reasoning.
A better approach: let the model decide first, then ask "what is your decision based on?" and verify whether the claimed basis actually has causal influence. This is closer to real legal reasoning — judges form an initial judgment, seek legal grounds, then test whether the grounds truly support the judgment.
The value of counterfactual auditing lies exactly here: it asks not "what did the model say?" but "did what it said affect its decision?" That is a paradigm shift from "citation accuracy" to "citation causal power."
---
Paper Links
One-line summary: When LLMs cite legal provisions, citation rates approach 100% but causal reliance sits near 50% — citations are rationalization shells, not links in the decision chain. Counterfactual auditing (swap the citation, see if the verdict changes) is the right way to test citation faithfulness.