AI Search Has a Citation Problem (CJR, March 2025)

Source: https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php

Overview

The Tow Center for Digital Journalism at Columbia Journalism Review published a study in March 2025 comparing eight AI-powered search engines on one core capability: correctly citing news articles used to generate answers. The conclusion of the headline says it directly — all of the tested engines were bad at citing news.

Key points

• The researchers ran over 3,000 individual queries against eight AI search products, asking about excerpts of news articles whose true source was known.
• Collectively, the engines returned incorrect citations in roughly 60 percent of generations — meaning the majority of answers failed to correctly attribute the news content they drew from.
• No engine was reliably accurate. The best-performing product still produced incorrect citations in a large fraction of responses, while the worst performer misattributed sources in nearly every generation tested (widely reported as ~94% incorrect for Grok-3, with Perplexity Pro the best of a weak field at ~37% incorrect).
• Common failure modes observed:
• Fabricated or hallucinated citations — URLs or articles that did not exist;
• Confusion between publishers — attributing one outlet's content to another;
• Overconfident wrong answers even when correct excerpts were provided in the prompt;
• Inconsistency: the same engine could cite the same content correctly in one run and wrongly in another.
• The authors stress the asymmetry of the problem: AI search products present answers and citations as authoritative and trustworthy, so citation errors undermine user trust, harm publishers whose traffic and attribution are displaced, and can launder misinformation.

Why it matters

AI search (RAG-style retrieval-augmented answer engines) is increasingly a gateway to news. If the citation layer — the mechanism meant to connect answers to verifiable sources — is broken in most responses, the end-to-end promise of these systems fails on a basic accountability dimension.

Recommendations from the study

• Vendors should treat citation accuracy as a core quality metric, not a cosmetic feature.
• Systems should abstain or flag uncertainty when sources cannot be reliably identified.
• Independent, repeatable evaluation of citation behavior should inform press coverage and consumer trust decisions.

Relation to broader search/retrieval evaluation

This study complements IR literature on retrieval-augmented generation (RAG) evaluation: it measures a *process* metric — citation/attribution correctness — rather than answer-only metrics like nDCG or task success. It is a useful empirical reference point alongside RAG evaluation frameworks (e.g., ARES, RAG-survey literature) that argue generation quality cannot be assessed without verifying grounding and provenance.

Note: Detailed per-engine tables and methodology should be checked against the original CJR article at the source link above.