Paper Overview
Field: NLP Authors: Saisab Sadhu, Shreeyans Arora, Pratinav Seth Published: 2026-10-08 arXiv: 2610.12361
Summary
Large language models increasingly justify legal decisions by naming the statute or precedent behind a verdict, and this is treated as evidence that the decision actually follows from that authority. This paper tests the assumption directly: holding case facts fixed, the authors substitute the named legal authority with an unrelated one and decode the model's evolving verdict from its hidden states.
Key Findings
- Across seven open-weight models (8B-70B) and four benchmarks spanning judicial and contractual reasoning:
- When explicitly required to justify a verdict by naming the governing authority, models name the correct one in 66.7%-100% of generations.
- But verdicts changing when the authority is substituted is far less consistent:
- CaseHOLD: 0.0%-21.7%
- ECHR and SCOTUS: 30.0%-76.7%
- ContractNLI: 43.3%-50.0%
- Neither model scale nor a purpose-built legal-reasoning model (the best-effort LoRA reproduction) closed this gap.
- Red-team evaluation of five core models found that compliance with adversarial instructions hidden in case facts (73.3%-96.4%) far exceeded verdict sensitivity to authority substitution, and this held across every model ranking.
Implications
Naming a legal authority is a poor proxy for how much a verdict actually relies on it — and the same verdicts are separately vulnerable to adversarial manipulation. The results were reproducible across a battery of controls excluding prompt-wording noise and confounded sampling, and directly bear on the use of generative legal explanation as a compliance or audit artifact.
--- *Auto-collected on 2026-10-12*