Key points
A 5-page paper (arXiv 2605.04908, published 2026-05-06) by Łukasz Kidziński and Kevin Thomas delivers a stark verdict: general-purpose frontier LLMs with web search miss most of the long-tail knowledge in specialized domains.
- The benchmark pitted Gosset — a curated pharma asset database with a chat interface — against Claude Opus 4.7 + web, GPT 5.5 + web, Gemini 3.1 Pro + web, and Perplexity sonar-pro on 10 niche oncology/immunology targets: TL1A, OX40L, IL-36R, TROP-2, B7-H3, ROR1, NaPi2b, Claudin 18.2, FAP, GPRC5D.
- These targets are dominated by long-tail pipelines from small biotechs, Chinese developers, and academic projects with little presence in mainstream media.
- Title: *Curated AI beats frontier LLMs at pharma asset discovery*
- arXiv ID: 2605.04908
- Published: 2026-05-06
- Authors: Łukasz Kidziński, Kevin Thomas
- Length: 5 pages, 5 figures, 1 table
- Systems compared: Gosset (curated drug-asset index) vs Claude Opus 4.7 + web_search (20 budget) vs GPT 5.5 + web_search_preview (20 budget) vs Gemini 3.1 Pro + Google Search grounding (20 budget) vs Perplexity sonar-pro (high context)
- Targets: TL1A, OX40L, IL-36R, TROP-2, B7-H3, ROR1, NaPi2b, Claudin 18.2, FAP, GPRC5D
- Headline data: Gosset 451 verified drugs vs GPT+web 140 (3.2x); Gosset precision 1.000 / recall 100% vs union of 451; Claude+web 1 hallucination, Perplexity 2
- Architecture: curated index exposed as MCP server for tool calling by any frontier LLM
Results (aggregate, human-expert verified)
| System | Verified Drugs | Precision | Recall (vs union) | Hallucinations | |:-------|:--------------:|:---------:|:-----------------:|:--------------:| | Gosset (curated) | 451 | 1.000 | 100% | 0 | | GPT 5.5 + web | 140 | 1.000 | ~31% | 0 | | Gemini 3.1 Pro + web | ~120 | 1.000 | ~27% | 0 | | Claude Opus 4.7 + web | ~110 | 0.992 | ~24% | 1 | | Perplexity sonar-pro | ~80 | 0.975 | ~18% | 2 |
Gosset returned 451 verified drugs; the best frontier system (GPT + web) returned 140 — less than a third. For competitive intelligence in pharma, this means GPT missed roughly two-thirds of competitor pipelines.
What counts as a "verified drug"?
The paper used a three-layer verification pipeline: 1. Deterministic auto-pass — drugs with active clinical trials, FDA approval, or sponsor commitments. 2. Three-AI-judge cross-check — Claude, GPT, and Gemini independently evaluate; 2-of-3 majority gives a preliminary verdict. 3. Human expert sign-off — a reviewer with pharma pipeline background audits all edge cases.
All figures reflect post-review status, not raw system outputs.
Late-stage parity, long-tail collapse
On widely reported approval/Phase 3 anchor drugs, all five systems performed equally — frontier LLMs can find press releases. But a typical target's preclinical and early-clinical pipeline is roughly 10x the length of its late-stage anchors, and web-indexed sources cover this long tail sparsely.
This is an indexing-coverage gap, not a reasoning gap. If a Chinese biotech's Claudin 18.2 pipeline appears only on a conference poster, never covered by English media, it simply doesn't exist to a web-grounded system.
Hallucinations in enumeration tasks
Claude + web produced 1 hallucinated drug (precision 0.992); Perplexity 2 (precision 0.975). Absolute counts are small but the direction is concerning: asking generative models for exhaustive named-entity lists activates hallucination tendencies even with retrieval augmentation. Errors include misattributed molecules (rocatinlimab was labeled OX40L when it binds the OX40 receptor) and canonical-name collisions (different molecules sharing developer codes like IMB101 or SAR446309). A structured database cannot hallucinate a nonexistent entry.
Latency
Gosset responds sub-second — a single MongoDB query round-trip. The frontier systems require multiple web searches plus generation, roughly two orders of magnitude slower — the difference between interactive and wait-and-see workflows.
The key insight: MCP changes everything
Gosset exposes its curated asset index as an MCP (Model Context Protocol) server — any frontier model can call it like a web search tool. The LLMs don't need to be replaced; only the data source changes. Same prompts, same natural language understanding, same reasoning — just swap "read the open web" for "read a curated index." MCP is Anthropic's 2024 open standard for connecting AI systems to external data sources and tools through unified interfaces.
The uncomfortable question
If your enterprise uses GPT/Claude/Gemini for competitive intelligence, market research, or due diligence, are your answers covering only the third of the landscape that got news coverage? Decisions built on these outputs rest on a ~69% blind spot.
The paper's real contribution is not "Gosset is good" — it provides a quantifiable controlled experiment proving curated indexes overwhelmingly beat generic web search on the professional long tail. This is not a race about model size; it is a race about information infrastructure.
Curated indexes won't replace LLMs — but they are redefining the battlefield.
---