Reading Is Not Using: When AI "Reads" But Doesn't "See"
> *"We don't see things as they are, we see them as we are."* — Anaïs Nin
This post is a detailed walkthrough of the paper: Liu, M., & Liu, Z. (2026). *Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows*. arXiv:2608.24842.
The Opening Parable
Imagine hiring a private detective to investigate a company's finances. He reads ten years of annual reports and produces a detailed report full of negative signals: high debt, tight cash flow, fierce competition. Yet when asked for a recommendation, he says "invest"—and when asked why, cites facts about an entirely different company. He retrieved everything correctly, but never integrated the information into his judgment.
This mirrors the paper's core finding: LLMs can accurately retrieve information, but that information may not influence their judgments.
Chapter 1: The Retrieval–Integration Gap
The Core Finding
> As irrelevant context grows from 2,000 to 128,000 tokens, the effect of risk disclosures on investment decisions falls to experimental noise—even though direct retrieval remains accurate.
- 2,000 tokens ≈ a short article (~1,500 English words)
- 128,000 tokens ≈ a small book (~90,000 English words)
- Noise level = the effect is statistically indistinguishable from zero
- Across model families (GPT, Claude, Llama, etc.)
- Across judgment tasks (buy/sell recommendations, risk assessment, valuation)
- Even in controlled experiments with synthetic text replacing real 10-K reports
- GPT-3.5 breaks down around 8,000 tokens
- GPT-4 holds out to 32,000 tokens
- But even the strongest models fail at 128,000 tokens
- Hallucination: fabricating information that doesn't exist
- Read but not used: retrieving correct information but not applying it
- Liu, M., & Liu, Z. (2026). *Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows*. arXiv:2608.24842.
- Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. *Review of General Psychology*, 2(2), 175–220.
- Kahneman, D. (2011). *Thinking, Fast and Slow*. Farrar, Straus and Giroux.
- Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. *Advances in Neural Information Processing Systems*, 33, 9459–9474.
An AI can correctly locate a supply-chain risk disclosure on "page 47" of a 10-K, yet its final investment recommendation is completely unaffected by it.
It's Systemic, Not a Bug
The phenomenon holds:
Chapter 2: Measuring "Read But Not Used"
Experimental Design
1. Fix the core information (target company's financials) 2. Vary only irrelevant context: 2K / 8K / 32K / 128K tokens of real financial text from other companies 3. Measure two things: direct retrieval ("what are the risk factors?") and judgment impact ("would you invest?")
Stronger Models Only Delay the Problem
A bigger cup holds more water—but still overflows eventually.
Causal Findings
Compression summaries and source-text lookup jointly transmit disclosure information into judgment. Like water pressure in a long pipeline, the signal decays at two points: (1) key information is diluted during summarization, and (2) even if preserved, it is "not found" or "not recalled" at decision time.
Chapter 3: Workflow Architecture Decides Success
The Chunk-and-Summarize Problem
The common pipeline (chunk → summarize each chunk → combine) acts like the telephone game: information leaks out at every hop, even when retrieval is accurate.
The Fix: Targeted, Structured Restatement
> A goal-directed, structured restatement placed near the decision restores the information's influence.
Instead of a 100-page report, make the detective list the 5 key facts directly relevant to the conclusion right before giving the verdict.
Joint Determination
> AI analyst performance is jointly determined by model capability and workflow architecture.
A strong model in a bad pipeline fails; a great pipeline on a weak model fails. Many deployments obsess over model choice while neglecting workflow design—yet the two may matter equally.
Chapter 4: Deeper Reflections
"Read But Not Used" vs. Hallucination
The latter is more insidious because the system appears to work—it answers retrieval questions correctly—while its judgments are disconnected from what it read.
Cognitive Science Angle
The phenomenon resembles human confirmation bias and selective attention: like counting basketball passes while missing who scored, information "enters the eyes" but is never processed.
Implications for RAG
> Retrieval-based evaluation can certify systems whose investment judgments ignore information they demonstrably retrieved.
RAG evaluation must not stop at retrieval accuracy; it must measure whether retrieved information actually shapes the output.
Epilogue: From "Reading" to "Using"
In high-stakes domains—medical diagnosis (an AI reads an allergy history but ignores it when prescribing), legal advice, financial analysis—the consequences of "read but not used" can be catastrophic. The bridge between information and judgment is not automatic; it must be built through deliberate workflow design. The paper's open challenge: how to design workflows so that retrieved information reliably influences final judgment?