Reading Without Using: How LLMs Retrieve Financial Risk Information but Ignore It in Investment Judgments

Forum topic · 小凯 · 2026-08-26

Summary

A new paper uncovers a troubling phenomenon in AI financial analysts: large language models can accurately retrieve information from long corporate filings, yet that retrieved information fails to influence their judgments. Experiments show that when irrelevant context grows from 2,000 to 128,000 tokens, the effect of risk disclosures on investment decisions drops to noise levels, even though direct retrieval remains accurate. The gap is consistent across GPT, Claude, and Llama families, and across buy/sell, risk, and valuation tasks. Causal analysis identifies compression and source-lookup stages as the points where influence decays. Common chunk-and-summarize pipelines amplify the problem by stripping context near the decision point. The authors show that a targeted, structured restatement placed adjacent to the decision restores informational impact. They argue AI analyst performance is jointly determined by model capability and workflow architecture, with implications for medical, legal, and financial applications using RAG systems.

Core Finding

The paper *Reading Is Not Using* documents a retrieval-judgment gap in LLM-based financial analysts: models accurately locate risk disclosures in long filings, but those disclosures do not move their final investment recommendations. When unrelated context scales from 2,000 to 128,000 tokens, the influence of risk information on judgments falls to noise-level, while retrieval accuracy remains intact. The effect holds across GPT, Claude, and Llama families and across buy/sell, risk-assessment, and valuation tasks, indicating a structural property of current LLMs rather than a model-specific bug.

Key Points

Retrieval is not the bottleneck. Standard RAG evaluations measure whether a system can find the right passage. This paper argues such evaluations can certify systems whose final judgments contradict the passages they retrieved.
Scale delays but does not solve the problem. GPT-3.5 shows the gap near 8,000 tokens; GPT-4 holds longer, around 32,000 tokens; even the strongest model collapses at 128,000 tokens. A larger cup still overflows.
Two failure points identified by causal intervention. Compression (long text to summary) and source-lookup (re-accessing original text at decision time) lose informational pressure, so the signal that reaches the final judgment is diluted.
Chunk-and-summarize pipelines worsen the gap. Sequential summarization acts like a telephone game: each pass discards detail, and by the final step the decision-relevant facts may be absent from the prompt that produces the recommendation.
A targeted, structured restatement recovers influence. Placing a decision-specific, schema-formatted restatement directly before the recommendation step restores the effect of retrieved information on the output.
Architecture and capability jointly determine reliability. Strong models with poor workflows still fail; weak models with good workflows also fail. The paper argues workflow design deserves as much engineering attention as model selection.

Implications Beyond Finance

The same pattern is likely in medical, legal, and other high-stakes RAG deployments, where a system may answer questions about a document correctly yet produce a recommendation that ignores it. Evaluation frameworks must move past retrieval accuracy and measure whether retrieved content causally affects outputs.

References

Liu, M., & Liu, Z. (2026). *Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows*. arXiv:2608.24842.

Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. *NeurIPS*, 33, 9459–9474.

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. *Review of General Psychology*, 2(2), 175–220.