GPT-5 and Claude 4.5 as News Anchors: Six AI Chatbots Tested on Live News, With 90%+ Accuracy — But Three Hidden Flaws
Ask ChatGPT "what happened in India today?" and you get one answer. Ask Grok the same question and you get another. Which one is correct? Are these models actually "reading the news," or just making things up?
A team from Stanford University and multiple institutions did something unprecedented: over 14 days (February 9–22, 2026), they extracted daily news from six BBC News regional services (Americas/Canada, Arabic, Africa, Hindi, Russian, Turkish), built 2,100 factual questions, and had six state-of-the-art AI chatbots answer them — 12,600 tests in total.
The six contenders: Gemini 3 Flash, Gemini 3 Pro, Grok 4, Claude 4.5 Sonnet, GPT-5, and GPT-4o mini. An all-star lineup.
The Surface Results: AI News Capabilities Are Impressive
The best systems achieved over 90% accuracy in multiple-choice mode — genuinely impressive for events that happened only hours earlier. Previous real-time QA benchmarks showed far weaker performance. AI has made a qualitative leap in "reading the news."
But that was multiple choice. When researchers switched to free-response mode, every model's accuracy dropped 11–13 percentage points. Multiple choice gives models hints (the answer is among the options), while free response reflects how real users actually ask questions.
Flaw #1: A Hidden Bias Against Hindi
All models performed significantly worse on Hindi news — only 79% accuracy, versus 89–91% for other languages. Why?
The researchers found a disturbing pattern: when answering Hindi questions, models cited English Wikipedia most often — not Hindi-language news outlets. In other words, models were "translating" the question, searching in English, and answering Hindi questions with English sources. The facts reported could differ entirely from what local news actually covered.
It's like asking a French journalist "what happened in Paris today?" and having them consult English Wikipedia, then describe Paris from an English-language perspective — the information may not be wrong, but it's not the Paris locals care about.
Flaw #2: 70% of Errors Come From "Not Finding," Not "Not Understanding"
This is the most counterintuitive finding. When researchers classified all errors, over 70% were retrieval failures — the model failed to find the right source. When the model did find the correct source, it almost always extracted the correct answer.
This means: AI reasoning is already strong enough; the bottleneck is search and retrieval infrastructure. Models aren't failing to "understand the news" — they're failing to "find the news." This contradicts common intuition: we assume AI errors come from insufficient comprehension, but in reality, it's more like a smart person who doesn't know how to use a library.
Flaw #3: One "Trap Question" Can Crash Accuracy
The researchers designed "adversarial questions" — embedding subtle false premises into normal questions. For example, if the news says a country's president visited France, the question asks what the president said while visiting Germany.
The results were shocking: accuracy plummeted from 88–96% to 19–70%. The most vulnerable model accepted the false premise 64% of the time, fabricating answers on top of misinformation.
Even more subtle was a "detection–accuracy paradox": the model best at detecting false premises ranked only second on adversarial accuracy, while a weaker detector ranked first. Strong detectors tend to "refuse to answer when they detect a problem" — but refusal still counts as an error, because sometimes you must not only spot the flawed premise but also recover from it and provide the correct information.
Why Should You Care?
The paper reveals a deeper problem: high headline accuracy can mask systematic inequality, absolute dependence on retrieval, and fragility to the way real users ask questions.
When you see a headline claiming "AI news accuracy exceeds 90%", ask three questions:
1. Was it multiple choice or free response? (A 10+ point gap) 2. What language was it in? (Hindi may lag by 10 points) 3. Did questions contain misleading premises? (Potentially a 50+ point gap)
AI is becoming how hundreds of millions of people access news. This paper shows these systems perform better than we might expect — but are also more fragile than we imagine. The greatest danger isn't that they err; it's that their errors are so systematic and so subtle that users can hardly detect them.
---
Paper: Evaluating Commercial AI Chatbots as News Intermediaries