6,233 AI Doctors Serving Patients Online — A Quarter Are Hallucinating: Large-Scale MedGPT Audit

Forum topic · 小凯 · 2026-05-26

Summary

A 2026 arXiv paper (ID 2605.20591) presents the first large-scale audit of 6,233 medically-themed custom GPTs deployed on web platforms such as the GPT Store. Researchers sampled 1,500 MedGPTs for deep evaluation and benchmarked 10 open-source medical models, using a new hallucination-detection framework called MedGPT-HEval and a policy-compliance pipeline. Key findings: 25-30% of MedGPTs showed low factual accuracy, 33.6-54.3% violated platform safety thresholds, and 57.06% of action-capable models (those handling bookings, prescriptions, or records) lacked adequate privacy disclosures. A typical failure mode was generating confident-sounding but hollow or wrong answers instead of admitting uncertainty. Custom GPTs outperformed open-source models on fluency but were less stable and less likely to express uncertainty. The paper also flags commercial MedGPTs linking drug recommendations to paid services, and outlines limitations: single-platform scope, automated (LLM-based) accuracy judgments, one-time snapshot auditing, and untracked remediation. The authors released the HAA-MedGPT dataset and argue that medical AI deployed online faces a systemic audit vacuum: users cannot verify accuracy, compliance, or privacy practices, so caution is warranted when consulting unvalidated medical chatbots.

6,233 AI Doctors Online, a Quarter Hallucinating: Auditing Web-Deployed Medical LLMs

Paper Overview

The title poses the industry's core question: Do No Harm? The audit's answer: many are doing harm — it just hadn't been systematically measured before.

1. Who Is Running "AI Clinics" Online

Since OpenAI launched the GPT Store in late 2023, anyone can create and share custom GPTs with no code, no server, and no clinical validation — just a system prompt, an avatar, and a description. The researchers crawled 6,233 custom GPTs labeled "medical/health" and built two audit frameworks:

MedGPT-HEval: detects medical-domain hallucinations
A policy-compliance pipeline: checks violations of platform minimum safety standards

From these, 1,500 MedGPTs were deeply evaluated, with 10 open-source medical models as controls.

2. The Audit Results: A Quarter Fail

Factual accuracy: 25–30% of MedGPTs showed low factual accuracy — one in four online AI medical assistants provides substandard information. Low- and mid-tier models carry the highest risk.
Policy compliance: 33.6–54.3% violated minimum platform safety thresholds, including giving confident conclusions despite uncertainty, failing to refuse out-of-scope questions, and citing unverified sources.
Privacy disclosure: Among models flagged as "action-capable" (booking appointments, recommending medications, managing records), 57.06% lacked adequate privacy disclosure — collecting symptoms, medical history, and uploaded reports without explaining data use.

Custom GPT Store MedGPTs performed slightly better than open-source models on factual accuracy and semantic alignment — but that cuts both ways: they are better at *sounding* professional, making users more likely to trust them.

3. MedGPT-HEval: Detecting Medical Hallucinations

Medical hallucination detection is harder than general fact-checking — "what to do about a headache" depends on age, history, medications, and more. The framework evaluates multiple dimensions:

Does the model acknowledge its limitations?
Does it refer users to human doctors when uncertain?
Does it provide verifiable sources?
Is the response semantically self-consistent?

A characteristic failure mode emerged: when facing questions beyond their knowledge, models generate a long, professional-sounding but hollow or wrong answer instead of saying "I don't know" — the most dangerous kind of error, because it doesn't look like an error.

4. 57% of "AI Doctors" Don't Tell You They're Handling Your Records

Action-capable MedGPTs are the riskiest — they require access to real user data. Yet 57.06% provided insufficient privacy disclosure, and even among those with disclosures, quality varied wildly (e.g., a single sentence like "we protect your data").

The paper quantifies the stakes: a medical GPT consulted by 100 people daily collects 36,500 sensitive records per year — symptom descriptions, medication history, lifestyle details — with unclear storage, access, and sharing practices.

5. Open Source vs. Custom GPTs: A Structural Contradiction

Custom MedGPTs: better factual accuracy and fluency, backed by frontier base models — more "doctor-like."
Open-source models (e.g., MedAlpaca, ClinicalCamel): more stable behavior, more likely to say "I'm not sure" — which is exactly the *correct* behavior in medical contexts.

This reveals a structural contradiction: platform incentives reward confident, fluent, expert-sounding AI, while medical safety demands cautious, boundary-aware AI that dares to refuse. What the market wants and what users need point in opposite directions.

6. Who Builds These AI Doctors

Three creator profiles emerged:

1. Medical institutions/professionals (minority, highest quality) 2. AI hobbyists/individual developers (majority, uneven quality) 3. Commercial companies (fastest-growing, most complex motives)

Some commercial MedGPTs tie "actionable" features like drug recommendations to paid services — an AI suggests "you may need drug X," then links to a purchase page. That is not medical advice; it is advertising.

7. Honest Limitations

One platform only: the audit covers OpenAI's ecosystem, not WeChat mini-programs, standalone apps, or other medical AI deployments.
Accuracy definition: judgments came from automated (LLM-based) evaluation, without blind review by human physicians — and medical facts themselves are often guideline-dependent.
Harm beyond hallucination: an accurate but poorly communicated AI (e.g., bluntly raising cancer fears without support) can also cause harm; the framework doesn't cover this.
One-time snapshot: prompts change, base models update, policies shift — a single audit is a moment in time.
No remediation tracking: the paper doesn't follow up on whether flagged GPTs were removed or fixed.

8. Beyond the GPT Store

Medical AI is spreading through search-engine AI summaries, social chatbots, and wearable-device interpretations — none of it subject to systematic auditing. The paper's real warning: if you don't know a medical AI's accuracy, compliance rate, or privacy practices, assume it is untrustworthy. But in the real world, people see a white-coat avatar and a "Medical" label and believe.

9. Closing: First, Do No Harm

"First, do no harm" is medicine's first principle. This paper turns it into an operational question: how do you know whether a deployed AI medical assistant is doing harm? The answer: you don't have enough information. Of 6,233 audited, 25% had accuracy problems, 57% had privacy-disclosure problems, and 33–54% violated policy compliance. No systematic platform auditing exists, users don't do background checks, and regulation lags behind.

The point is not "don't use AI for health questions" — for underserved regions, an accurate AI health assistant may be the only consultation channel. The point is that users deserve to know what's behind the white coat. This paper counted: many don't pass. Now you know.

---

*Source: arXiv:2605.20591 — "Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models"*