Overview
A May 2026 study by Łukasz Kidziński and Kevin Thomas (*Curated AI beats frontier LLMs at pharma asset discovery*, arXiv:2605.04908) presents a controlled head-to-head benchmark comparing a curated specialized index system (Gosset) against four frontier general LLM + web search systems on pharmaceutical pipeline intelligence tasks.
The core premise: in long-tail knowledge retrieval for specialized domains, index quality may be more decisive than model reasoning ability.
> Note: Gosset platform architecture > Gosset is an AI platform for the pharma industry built around a curated drug asset index database, with structured annotations across three dimensions: target, modality, and indication. Users submit queries through a natural language chat interface; the system parses them into structured filters and queries the index directly. Unlike frontier LLMs, Gosset has no live web access—its information comes from a pre-compiled proprietary dataset.
Experimental Design
The study ran controlled comparisons on 10 niche oncology/immunology targets:
- Targets: TL1A, OX40L, IL-36R, TROP-2, B7-H3, ROR1, NaPi2b, Claudin 18.2, FAP, GPRC5D
- Selection criteria: immunology/oncology diversity, medium attention, known significant long-tail pipelines
- A 3.2x gap in verified drug count between Gosset and the best frontier system (GPT + web)
- The gap is especially pronounced on long-tail targets (TL1A, B7-H3, NaPi2b, Claudin 18.2) with many preclinical and Asian-developer programs
- All systems performed identically on approved/Phase III anchor drugs (late-stage parity), confirming frontier LLMs retrieve high-visibility information well
- GPT + web and Gemini + web achieved perfect precision (1.000)
- Claude + web produced 1 hallucinated drug (precision 0.992); Perplexity produced 2 (precision 0.975)
- Hallucination types included canonical-name collisions (different molecules sharing a developer code) and wrong-target attribution (e.g., rocatinlimab mislabeled as OX40L when it actually binds the OX40 receptor)
- Title: *Curated AI beats frontier LLMs at pharma asset discovery*
- arXiv ID: 2605.04908
- Published: 2026-05-06
- Authors: Łukasz Kidziński, Kevin Thomas
- Length: 5 pages, 5 figures, 1 table
Five systems compared:
| System | Backend | Search capability | Output limit | |:-------|:--------|:------------------|:-------------| | Gosset | Proprietary parser + curated index | No live web | Max 200 rows | | GPT + web | GPT 5.5 (Azure) | web_search_preview, 20 budget | JSON schema | | Gemini + web | Gemini 3.1 Pro | Google Search grounding, 20 budget | JSON schema | | Claude + web | Claude Opus 4.7 (API) | hosted web_search, 20 budget | JSON schema | | Perplexity | sonar-pro | search_context_size="high" | JSON schema |
All systems received the same natural language prompt, required to return a JSON list of {name, sponsor, modality, phase, indication} with an explicit aliases field for alias-aware deduplication.
Quantitative Results
| System | Verified Drugs | Precision | Recall (vs 451 union) | Hallucinated | |:-------|:--------------:|:---------:|:---------------------:|:------------:| | Gosset | 451 | 1.000 | 100.0% | 0 | | GPT + web | 140 | 1.000 | 31.0% | 0 | | Gemini + web | ~120 | 1.000 | ~26.6% | 0 | | Claude + web | ~110 | 0.992 | ~24.4% | 1 | | Perplexity | ~80 | 0.975 | ~17.7% | 2 |
Key findings:
Hallucination analysis:
Latency: Gosset answered in sub-second time (a single database query); frontier systems were orders of magnitude slower due to multiple web searches plus generation.
Three-Layer Verification Pipeline
1. Deterministic Auto-Pass: Drugs with industry-grade evidence (active clinical trials, FDA approvals, sponsor commitments) auto-verified without LLM calls, clearing most uncontroversial cases. 2. Three-AI-Judge Cross-Check: Remaining cases independently evaluated by Claude Opus 4.7, GPT 5.5, and Gemini 3.1 Pro (all with web search) using LLM-as-a-judge; 2/3 majority yields verified / hallucinated / unsure verdicts. 3. Human Expert Sign-Off: A reviewer with pharma pipeline background audited all edge cases, handling canonical-name collisions, indirect pathway annotations, and wrong-target attributions. All published figures reflect post-review status.
> Note: Alias-aware union-find deduplication > To prevent systems from inflating counts via alias packing (e.g., counting "Tecotabart vedotin (LM-302, TPX-4589, BMS-986476)" as multiple entries), each system was required to return explicit alias lists, and an alias-aware union-find algorithm merged drugs whose alias sets shared any non-trivial member.
Indexing Problem vs. Model Problem
The paper's central argument: the recall gap is not a model quality issue—it is an indexing issue.
Supporting behavioral evidence: 1. All frontier systems handled structured output, NL understanding, and multi-step reasoning normally 2. Late-stage parity: all systems correctly retrieved highly visible approved/Phase III drugs 3. The gap concentrated in preclinical and early-clinical programs—found in sponsor websites, conference posters, patents, and Chinese-language materials, but inadequately indexed by search engines
This has important implications for the Model Context Protocol (MCP): Gosset exposes its curated index as an MCP server, allowing frontier LLMs to switch information sources via tool calls without any change to themselves.
When index coverage has structural gaps, improvements in model reasoning cannot compensate for recall deficits.
Practical Recommendations
1. Distinguish processing length from recall coverage: "1M token context" measures input handling, not how many facts can be recalled. Measure effective recall separately for exhaustive enumeration tasks. 2. Assess information source blind spots: Before using general LLMs for competitive intelligence, quantify what fraction of the target domain the open web covers and whether missing long-tail information (regional developers, academic projects, preprints) affects decision quality. 3. Consider curated indexes as MCP tools: Wrapping a curated index as an LLM-callable tool (rather than replacing the LLM) preserves reasoning ability while fundamentally improving retrieval—likely the optimal hybrid architecture for specialized domains. 4. Build domain-specific verification pipelines: The three-layer architecture (auto-pass → AI cross-check → human expert) is a general template for high-stakes domain applications.
Limitations and Future Directions
Limitations: ① Target selection may favor domains where Gosset's data is rich; gaps would likely be smaller for broadly covered targets (e.g., PD-1, HER2). ② The verification judges are themselves LLMs with inherited calibration errors. ③ Absolute pipeline recall is unmeasurable—the cross-system union reflects only the discoverable universe, not the true full pipeline.
Future work: ① Benchmarking frontier LLMs combined with the Gosset MCP server (upcoming follow-up); ② Extending the framework to other specialized domains (legal precedent, patent landscapes, supply chain intelligence); ③ Developing automated curated index construction pipelines to lower production costs.
---