GPT-5 and Claude 4.5 as News Anchors: 6 AI Chatbots Score 90%+ on News QA — But Hide Three Major Risks
Ask ChatGPT "what happened in India today" and you get one answer. Ask Grok the same question and you get another. Which is right? Are these models actually "reading the news," or just making things up?
A team from Stanford and several other institutions did something unprecedented: over 14 days (February 9-22, 2026), they extracted daily news from BBC News' six regional services (US/Canada, Arabic, Africa, Hindi, Russian, Turkish) and constructed 2,100 factual questions, then had six state-of-the-art AI chatbots answer them — 12,600 tests in total.
The six "contestants": Gemini 3 Flash, Gemini 3 Pro, Grok 4, Claude 4.5 Sonnet, GPT-5, and GPT-4o mini. An all-star lineup.
Surface Results: AI News Capabilities Are Impressive
The best systems achieved over 90% accuracy in multiple-choice mode — genuinely impressive for events that happened just hours earlier. Previous real-time QA benchmarks showed far weaker performance. AI has made a qualitative leap in "reading the news."
But that was multiple choice. When researchers switched to free-response mode, all models' accuracy dropped 11-13 percentage points. Multiple choice gives models hints (the answer is among the options); free response reflects how real users actually interact.
Risk #1: A "Hidden Discrimination" Against Hindi
All models performed noticeably worse on Hindi news — only 79% accuracy, versus 89-91% for other languages. Why?
The researchers found a troubling pattern: when answering Hindi questions, the models' most-cited source wasn't Hindi-language news outlets, but English Wikipedia. In other words, the models were "translating" the question, searching in English, and answering Hindi questions with English sources. The facts they reported could differ entirely from what local news actually covered.
It's like asking a French journalist "what happened in Paris today" and having them consult English Wikipedia, then describe Paris through an English lens — the information may not be wrong, but it's definitely not the Paris locals care about.
Risk #2: 70% of Errors Come From "Can't Find It," Not "Misunderstanding"
This is the most counterintuitive finding. When the researchers classified all errors, over 70% were retrieval failures — the model failed to locate the correct information source. When the model did find the right source, it almost always extracted the correct answer.
This means: AI reasoning ability is already strong enough; the bottleneck lies in search and retrieval infrastructure. Models don't "fail to understand the news" — they "can't find the news." This contradicts common intuition. We assume AI errors stem from "insufficient understanding," but the model is more like a smart person who doesn't know how to use a library.
Risk #3: One "Trap Question" Can Crash Accuracy
The researchers designed "adversarial questions" — embedding subtle false premises into normal questions. For example, if the news said "a country's president visited France," the question would ask "what did the president say during his visit to Germany."
The result was shocking: accuracy plummeted from 88-96% to 19-70%. The most vulnerable model accepted the false premise 64% of the time, fabricating answers on top of the misinformation.
More subtly, the researchers found a "detection-accuracy paradox": the model best at detecting false premises only ranked second on adversarial accuracy, while a weaker detector ranked first. The reason: strong detectors tend to "refuse to answer once they detect a problem," but refusal counts as an error — sometimes you must not only spot the problem but recover from the false premise to give the correct information.
Why Should You Care?
The paper reveals a deeper problem: high headline accuracy can mask systematic inequality, absolute dependence on retrieval, and fragility to how real users ask questions.
When you see headlines like "AI news accuracy exceeds 90%," ask three questions: 1. Is this multiple choice or free response? (a 10+ percentage-point gap) 2. Which language? (Hindi may lag by 10 points) 3. Does the question contain a misleading premise? (potentially a 50+ point gap)
AI is becoming how hundreds of millions of people access news. This paper shows they do better than we expected — but are also more fragile than we imagined. The greatest danger isn't that they make mistakes; it's that their mistakes are so systematic and so hidden that users can hardly detect them.
---
Paper link: Evaluating Commercial AI Chatbots as News Intermediaries