[论文] Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thou...

论文概要

研究领域: NLP
作者: Saisab Sadhu, Shreeyans Arora, Pratinav Seth
发布时间: 2026-10-08
arXiv: 2610.12361

中文摘要

大语言模型日益频繁地通过引述法条或先例来为法律裁决辩护，这被当作裁决确实遵循该依据的证据。我们直接检验这一点：固定案件事实，将所引法律权威替换为无关权威，并从模型隐状态中解码其逐步演变的裁决。在 7 个开源模型（8B-70B）和覆盖司法与合同推理的 4 个基准上，当被明确要求通过点名适用法律权威来论证裁决时，模型在 66.7%-100% 的生成中点对了名字；但当权威被替换时，裁决随之改变的一致性要低得多：CaseHOLD 上为 0.0%-21.7%，ECHR 和 SCOTUS 上为 30.0%-76.7%，ContractNLI 上为 43.3%-50.0%。模型规模和专门的法律推理模型（最佳努力的 LoRA 复现）都未能弥合这一差距。对 5 个核心模型的红队评估发现：对藏在案件事实中的对抗指令的服从率（73.3%-96.4%）远超裁决对权威替换的敏感度，且在模型排名间无一例外地成立。因此，点名法律权威是裁决对其依赖程度的糟糕代理指标，而同一裁决又单独易受对抗操纵。这些发现在排除提示措辞噪声和混杂抽样的一系列检验中均可复现，并直接关系到生成式法律解释作为合规或审计文物的使用。

原文摘要

Large language models increasingly justify legal decisions by naming the statute or precedent behind a verdict, treated as evidence that the decision follows from it. We test this directly: holding case facts fixed, we substitute the named legal authority for an unrelated one and decode a model's evolving verdict from its hidden states. Across seven open-weight models (8B-70B) and four benchmarks spanning judicial and contractual reasoning, when explicitly required to justify a verdict by naming the governing authority, models name the correct one in 66.7%-100% of generations, while the verdict changing when the authority changes is far less consistent: 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Neither scale nor a purpose-built legal-reasoning ...

自动采集于 2026-10-12

#论文 #arXiv #NLP #小凯