Cited but Not Verified: Your AI's 'Trustworthy' Citations May Be Fabricating Facts

Forum topic · 小凯 · 2026-05-08

Summary

A PwC research team built an automated framework to audit citations in deep research reports generated by 14 major LLMs, evaluating each citation on three levels: link validity, content relevance, and factual accuracy. The results reveal a troubling gap: links work over 94% of the time and pages are relevant over 80% of the time, but only 39–77% of cited facts are actually supported by the source. Claude Opus 4.5 leads factual accuracy at 76.8%, while several OpenAI models achieve 100% task completion yet fail fact-checking nearly half the time. An ablation study shows that more search makes things worse: GPT-5.4's fact-check rate collapses from 78.6% with 2 tool calls to 16.7% at 150 calls due to information overload, conflation, misattribution, and over-inference. The paper argues that polished surface metrics create a false sense of trust. Practical advice: spot-check citations, monitor citation quality in agent pipelines, and treat AI-generated reports with skepticism regardless of how credible they look.

Cited but Not Verified: When AI Citations Look Right but Aren't

A post on zhichai.net reviews the PwC paper "Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents" (arXiv: 2605.06635) by Hailey Onweller, Elias Lumer, Austin Huber, Pia Ramchandani, Vamse Kumar Subbiah, and Corey Feld, from PwC's U.S. Commercial Technology and Innovation Office.

The author opens with a personal anecdote: while using Perplexity for medical research, they found that a cited source did not actually contain the number the AI attributed to it — data from a different article had been attached to the wrong citation.

Key points

1. The verification pyramid

The framework evaluates citations at three ascending levels of difficulty:

LinkWorks: does the URL return HTTP 200? Frontier models score >94%
Relevant Content: does the page discuss the claimed topic? >80%
Fact Check: does the source actually support the specific claim? Only 39–77%

The disconnect: links nearly always work and pages are usually relevant, but roughly half of factual claims may not be supported by their cited source.

2. Results across 14 models (130 research queries)

| Model | Task success | LinkWorks | Relevant | Fact Check | |---|---|---|---|---| | Claude Opus 4.5 | 90% | 98.7% | 95.7% | 76.8% | | Claude Haiku 4.5 | 83% | 98.9% | 91.1% | 68.9% | | GPT-5.2 | 100% | 98.3% | 92.3% | 58.8% | | Claude Sonnet 4.6 | 93% | 99.2% | 89.8% | 58.7% | | Codex | 100% | 96.9% | 91.9% | 54.1% | | Gemini 3.1 Pro | 90% | 94.1% | 80.7% | 48.5% | | GPT-5.4 | 100% | 100% | 93.7% | 47.7% | | GPT-5 Mini | 100% | 99.3% | 87.4% | 38.9% | | Llama 4 Maverick | 30% | 80.8% | 60.6% | 34.3% | | OSS-120B | 40% | 83.9% | 68.7% | 24.4% |

Notable findings:

Anthropic leads on factual accuracy. Claude Opus 4.5 hits 76.8% Fact Check — selective citation appears more effective than exhaustive citation.
OpenAI's 100% task success is a trap. GPT-5.4, GPT-5.2, Codex, and GPT-5 Mini always produce reports, but 40–60% of their factual citations fail verification.
Open-source models struggle at generation itself. Llama 4 Maverick (30%) and Pixtral Large (17%) often omit citations or produce unparseable output.
GPT-5.4 scores lower on Fact Check than GPT-5.2 (47.7% vs 58.8%) — more tool calls, worse accuracy.

3. Searching more makes accuracy worse

An ablation study on GPT-5.4 and Claude Opus 4.6 varied tool calls from 2 to 150:

| Tool calls | GPT-5.4 Fact Check | Claude Opus 4.6 Fact Check | |---|---|---| | 2 | 78.6% | 80.0% | | 30 | 43.0% | 69.2% | | 100 | 37.2% | 58.7% | | 150 | 16.7% | 57.9% |

The paper attributes this to information overload: conflation (mixing facts across sources), misattribution (crediting the wrong source), and over-inference (reading claims the source doesn't explicitly support). LinkWorks and Relevant Content stay above 92% throughout — only Fact Check degrades, meaning the problem is synthesis, not retrieval.

4. Plausible-looking errors are the most dangerous

The paper's ethics statement warns that high link-validity and relevance scores may create a false sense of trust. The hardest-to-detect citation hallucinations are exactly those this framework targets: a valid, relevant link whose content doesn't support the AI's specific claim.

5. Limitations of the framework

LLM-as-a-judge bias: Fact Check and Relevance are scored by another LLM; human calibration covered only 50–100 samples per model.
Web content drift: pages change over time, so numbers may vary on re-runs.
Binary judging: the framework doesn't distinguish factual error from paraphrase or interpretive difference (e.g., "~2%" vs "2.1%").

6. Practical recommendations

| Audience | Action | |---|---| | Regular users | Don't relax because citations exist — spot-check 2–3 randomly | | System builders | Integrate citation-quality monitoring into agent pipelines, not just surface metrics | | Researchers | Keep building automated fact-verification frameworks, especially for multi-source synthesis |

Paper details

Title: Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
Authors: Hailey Onweller, Elias Lumer, Austin Huber, Pia Ramchandani, Vamse Kumar Subbiah, Corey Feld
Institution: PwC, U.S. Commercial Technology and Innovation Office
arXiv: 2605.06635 (2026-05-07, cs.CL)
Link: https://arxiv.org/abs/2605.06635

The closing argument: when an AI report's links all work and pages are all relevant, we are trusting that the AI makes us *feel* it found correct information — not that it actually did. Skepticism toward AI citations is not distrust of technology; it is respect for facts.