GPT-5 and Claude 4.5 as News Anchors: 90%+ Accuracy, But Three Hidden Risks
Ask ChatGPT "what happened in India today" and you get one answer. Ask Grok the same question and you get another. Which is correct? Are these systems actually reading the news, or just making things up?
A team from Stanford and several partner institutions did something unprecedented: over 14 days (February 9–22), they extracted daily news from six regional BBC News services (US/Canada, Arabic, Africa, Hindi, Russian, Turkish), built 2,100 factual questions, and asked six state-of-the-art AI chatbots to answer them—12,600 tests in total.
The six contestants: Gemini 3 Flash, Gemini 3 Pro, Grok 4, Claude 4.5 Sonnet, GPT-5, and GPT-4o mini. An all-star lineup.
Surface Results: Impressive News Capabilities
The best systems achieved over 90% accuracy in multiple-choice mode—remarkable for events that happened only hours earlier. Prior real-time QA benchmarks showed far weaker performance. AI has made a qualitative leap in actually "reading the news."
But that was multiple choice. When researchers switched to free-response mode, every model's accuracy dropped 11–13 percentage points. Multiple choice gives models hints (the answer is in the options); free response reflects how real users actually interact with chatbots.
Risk 1: Hidden Discrimination Against Hindi
All models performed significantly worse on Hindi news—79% accuracy versus 89–91% for other languages. Why?
The researchers found a troubling pattern: when answering Hindi questions, the models' most-cited source was not Hindi-language media but English Wikipedia. In other words, models were "translating" the question, searching in English, and answering Hindi questions with English sources. The facts they reported could differ substantially from what local news actually covered.
It's like asking a French journalist "what happened in Paris today" and having them read English Wikipedia instead—technically not wrong, but not the Paris locals care about either.
Risk 2: 70% of Errors Are "Couldn't Find," Not "Misunderstood"
The most counterintuitive finding. After classifying all errors, the researchers found over 70% were retrieval failures—the model never found the correct source. When the model did find the right source, it almost always extracted the right answer.
This means AI reasoning is already strong enough; the bottleneck is search and retrieval infrastructure. Models aren't failing to understand the news—they're failing to find it. This contradicts common intuition: AI errors look like comprehension problems but are often more like a smart person who doesn't know how to use a library.
Risk 3: One "Trap Question" Can Crash Accuracy
Researchers designed adversarial questions embedding subtle false premises. For example, the news says a president visited France, but the question asks what the president said while visiting Germany.
The result was shocking: accuracy plummeted from 88–96% to 19–70%. The most vulnerable model accepted the false premise 64% of the time and confidently elaborated on misinformation.
Even subtler was a "detection–accuracy paradox": the model best at detecting false premises ranked only second on adversarial accuracy, while a weaker detector ranked first. The reason: strong detectors tend to "detect the problem and refuse to answer," but refusals count as errors—sometimes you must not only spot a false premise but recover from it and supply correct information.
Why Should You Care?
The paper reveals a deeper problem: high headline accuracy can mask systematic inequality, total dependence on retrieval, and fragility to how real users actually ask questions.
When you see headlines claiming "AI news accuracy exceeds 90%," ask three questions: 1. Multiple choice or free response? (A 10+ point gap) 2. Which language? (Hindi may lag by 10 points) 3. Does the question contain misleading premises? (A 50+ point gap)
AI is becoming how hundreds of millions of people access news. This paper shows these systems perform better than we expected—but are also more fragile than we imagined. The greatest danger isn't that they make mistakes; it's that their mistakes are so systematic and so hidden that users can hardly detect them.
---
Paper: Evaluating Commercial AI Chatbots as News Intermediaries