6,233 AI Doctors Online: A Quarter Give Unreliable Medical Advice

Forum topic · 小凯 · 2026-05-26

Summary

A 2026 arXiv paper (2605.20591), "Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models," presents the first large-scale audit of 6,233 medically-themed custom GPTs deployed on the GPT Store and similar platforms. The researchers sampled 1,500 MedGPTs for deep evaluation and benchmarked 10 open-source medical models, using a new hallucination-detection framework (MedGPT-HEval) and a policy-compliance pipeline. Key findings: 25-30% of MedGPTs show low factual accuracy; 33.6-54.3% violate platform safety thresholds; and 57.06% of "actionable" models (capable of booking, prescriptions, or records management) lack adequate privacy disclosures despite collecting sensitive health data. Custom GPTs outperform open-source models on fluency but are more prone to confident hallucination, while open-source models are more likely to admit uncertainty. The authors also document commercial conflicts of interest, such as drug recommendations tied to paid links. Limitations include GPT Store-only scope, automated (not clinician-blinded) accuracy judgments, and a single-point-in-time audit. The paper releases the HAA-MedGPT dataset and calls for systematic auditing of consumer-facing medical AI.

Overview

The paper "Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models" (arXiv: 2605.20591, submitted May 20, 2026; cs.CL, cs.CY) by Sunday Oyinlola Ogundoyin, Muhammad Ikram, and Rahat Masood is the first systematic, large-scale audit of medically-themed custom GPTs ("MedGPTs") deployed on the GPT Store and similar platforms.

Core contribution: Audit of 6,233 web-deployed medical GPTs, with a stratified deep evaluation of 1,500 samples and 10 open-source medical models
New artifacts: Hallucination-detection framework MedGPT-HEval and the HAA-MedGPT dataset

Key Findings

Factual accuracy: 25-30% of MedGPTs show low factual accuracy — roughly one in four online "AI doctors" provides substandard information, with low- and mid-tier models at highest risk.
Policy compliance: 33.6-54.3% of MedGPTs violate minimum platform safety thresholds, including giving confident answers when uncertain, failing to refuse out-of-scope questions, and citing unverified sources.
Privacy disclosure: Among "actionable" models (those that book appointments, recommend medications, or manage records), 57.06% lack adequate privacy disclosure — despite collecting sensitive data such as symptoms, medical history, and uploaded reports. The authors estimate a MedGPT consulted by 100 people daily could collect 36,500 sensitive records per year.
Typical failure mode: When facing questions beyond their knowledge, models generate long, professional-sounding but empty or wrong answers instead of admitting uncertainty — the most dangerous kind of error because it does not look like one.

Open-Source vs. Custom Models

Custom GPT Store MedGPTs score higher on factual accuracy and semantic fluency, backed by frontier base models — but their polished, confident tone makes users trust them more readily.
Open-source models (e.g., MedAlpaca, ClinicalCamel) are less fluent but more behaviorally stable, and more likely to say "I'm not sure" — which is the correct behavior in medical contexts.
The authors identify a structural contradiction: marketplace incentives reward confident, expert-sounding AI, while medical safety requires cautious, boundary-aware AI that refuses when appropriate.

Who Builds These AI Doctors

Developers fall into three groups: 1. Medical institutions/professionals (minority, highest quality) 2. AI hobbyists/indie developers (majority, uneven quality) 3. Commercial companies (fastest growing, most complex motives)

Some commercial MedGPTs tie "actionable" features like drug recommendations to paid links — behavior closer to advertising than medical advice. The paper does not accuse any single GPT but shows that a regulatory vacuum in medical AI is being rapidly filled by commercial interests.

Acknowledged Limitations

1. GPT Store only: The audit does not cover WeChat mini-programs, standalone symptom-checker apps, or other medical AI deployments. 2. Automated accuracy judgment: "Accuracy" is judged by an LLM-based framework without blinded clinician review, and medical facts are often ambiguous across guidelines. 3. Harm beyond hallucination: A fully accurate MedGPT can still harm via tone or lack of follow-up support; the framework does not measure communication-level harm. 4. One-time snapshot: Developer prompts, base models, and platform policies change; the audit reflects only a single moment. 5. No remediation tracking: The paper does not follow up on takedowns, platform responses, or developer fixes.

Broader Implications

The core warning extends beyond the GPT Store: search-engine AI overviews, social chatbots, and wearable AI all answer medical questions with similar audit gaps. The paper's practical takeaway — anchored in the Hippocratic principle *"First, do no harm"* — is that unless an AI medical assistant's accuracy, compliance, and privacy practices are known, it should be treated as untrustworthy by default. AI can still provide valuable health information, especially where doctors are scarce, but users deserve to know what is behind the white-coat icon.

---

*Source: arXiv 2605.20591 — "Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models"*