GPT-5 and Claude 4.5 as News Anchors: Top AI Chatbots Hit 90%+ Accuracy—but With Three Major Hidden Risks
Ask ChatGPT "what happened in India today" and you get one answer. Ask Grok the same question and you get another. Which one is right? Are these models actually "reading the news," or just making things up?
A team from Stanford University and several other institutions did something unprecedented: over 14 days (February 9–22, 2026), they extracted daily news from six regional BBC News services (US/Canada, Arabic, Africa, Hindi, Russian, Turkish), constructed 2,100 factual questions, and had six state-of-the-art AI chatbots answer them—12,600 tests in total.
The six contestants: Gemini 3 Flash, Gemini 3 Pro, Grok 4, Claude 4.5 Sonnet, GPT-5, and GPT-4o mini. An all-star lineup.
Surface Results: AI News Capabilities Are Impressive
The best systems achieved over 90% accuracy in multiple-choice mode—genuinely impressive for events that happened just hours earlier. In previous real-time QA benchmarks, models performed far worse. AI has made a qualitative leap in "reading the news."
But that was multiple choice. When the researchers switched to free-response mode, every model's accuracy dropped 11–13 percentage points. Multiple choice gives the model hints (the answer is in the options); free response reflects how real users actually interact.
Risk 1: Invisible Discrimination Against Hindi
All models performed significantly worse on Hindi news—79% accuracy vs. 89–91% for other languages. Why?
The researchers found a troubling pattern: when answering Hindi questions, the models' most-cited source was not Hindi news media but English Wikipedia. In other words, the models were "translating" the question, searching in English, and answering a Hindi question with English sources. As a result, the facts they reported could differ substantially from what local news actually covered.
It's like asking a French journalist "what happened in Paris today" and having them look up English Wikipedia, then tell you about Paris from an English-language perspective—the information may not be wrong, but it's definitely not the Paris locals care about.
Risk 2: 70% of Errors Are "Couldn't Find It," Not "Misunderstood It"
This is the most counterintuitive finding. After classifying all errors, the researchers found that over 70% were retrieval failures—the model failed to locate the correct source. When the model did find the right source, it almost always extracted the correct answer.
This means AI's reasoning capability is already strong enough; the bottleneck lies in search and retrieval infrastructure. Models aren't "failing to understand the news"—they "can't find the news." This contradicts common intuition: we assume AI errs because of limited comprehension, but in reality it's more like a smart person who doesn't know how to use a library.
Risk 3: One "Trap Question" Can Crash Accuracy
The researchers designed "adversarial questions"—normal questions embedding a subtly false premise. For example, if the news says "a country's president visited France," the question asks "what did the president say while visiting Germany."
The results were shocking: accuracy plummeted from 88–96% to 19–70%. The most vulnerable model accepted the false premise 64% of the time and ran with the misinformation.
Even more subtle: the researchers uncovered a "detection–accuracy paradox." The model best at detecting false premises ranked only second in adversarial accuracy, while a weaker detector ranked first. Strong detectors tend to "refuse to answer once a problem is detected," but refusal still counts as wrong—because sometimes you must not only spot the issue but also recover from the false premise and supply correct information.
Why Should You Care?
This paper reveals a deeper problem: high headline accuracy can mask systemic inequality, absolute dependence on retrieval, and fragility to how real users actually ask questions.
When you see headlines like "AI news accuracy exceeds 90%," ask three questions: 1. Was this multiple choice or free response? (a 10+ point gap) 2. What language? (Hindi may lag by 10 points) 3. Did the questions contain misleading premises? (potentially a 50+ point gap)
AI is becoming how hundreds of millions of people access news. This paper shows these systems do better than we expected—but are also more fragile than we imagined. The greatest danger isn't that they make mistakes—it's that they make mistakes so systematically and so invisibly that users can hardly notice.
---
Paper link: Evaluating Commercial AI Chatbots as News Intermediaries