Curated AI Beats Frontier LLMs at Pharma Asset Discovery: The Gosset Benchmark Study

Background: The Evaluation Blind Spot for Domain Knowledge Retrieval

General-purpose LLMs equipped with web search are now widely used for competitive intelligence, market research, and knowledge-intensive decision support. Yet existing evaluations focus mostly on general knowledge benchmarks (MMLU, GPQA, HumanEval) and rarely measure long-tail knowledge retrieval in specialized domains. In pharmaceutical asset discovery, the competitive landscape of a target is often composed of many preclinical and early-clinical programs from small biotechs, regional developers, and academic institutions—entities with extremely low visibility on the open web.

In May 2026, Kidziński and Thomas proposed a controlled head-to-head benchmark comparing a curated specialized index system (Gosset) against four frontier LLM + web search systems on pharma pipeline intelligence tasks. The study's core premise: for long-tail knowledge retrieval in specialized domains, index quality may matter more than model reasoning ability.

> 📌 Note: Gosset's architecture. Gosset is an AI platform for the pharmaceutical industry built around a curated drug asset index, with structured annotations across target, modality, and indication. Users query via natural-language chat; the system parses queries into structured filters and queries the index directly. Unlike frontier LLMs, Gosset has no live web access—its information comes from a pre-compiled proprietary dataset rather than the open web.

Experimental Design: A Controlled Head-to-Head Comparison

Target selection (diverse immunology/oncology targets with moderate attention and known significant long-tail pipelines):

• TL1A, OX40L, IL-36R, TROP-2, B7-H3, ROR1, NaPi2b, Claudin 18.2, FAP, GPRC5D

Five-system comparison:

| System | Backend | Search capability | Output constraint | |:-------|:--------|:------------------|:------------------| | 💎 Gosset | Proprietary parser + curated index | No live web | Max 200 rows | | 🤖 GPT + web | GPT 5.5 (Azure) | web_search_preview, 20 budget | JSON schema | | 🔮 Gemini + web | Gemini 3.1 Pro | Google Search grounding, 20 budget | JSON schema | | 🧠 Claude + web | Claude Opus 4.7 (API) | hosted web_search, 20 budget | JSON schema | | 🔍 Perplexity | sonar-pro | search_context_size="high" | JSON schema |

All systems received the same natural-language prompt and were asked to return a JSON list of {name, sponsor, modality, phase, indication}, explicitly including an aliases field to support alias-aware deduplication.

Quantitative Results: The Recall Gap

After the three-layer verification pipeline:

| System | Verified Drugs | Precision | Recall (vs 451 union) | Hallucinated | |:-------|:--------------:|:---------:|:---------------------:|:------------:| | 💎 Gosset | 451 | 1.000 | 100.0% | 0 | | 🤖 GPT + web | 140 | 1.000 | 31.0% | 0 | | 🔮 Gemini + web | ~120 | 1.000 | ~26.6% | 0 | | 🧠 Claude + web | ~110 | 0.992 | ~24.4% | 1 | | 🔍 Perplexity | ~80 | 0.975 | ~17.7% | 2 |

Recall gap analysis:

• A 3.2× gap in verified drug counts between Gosset and the best frontier system (GPT + web)
• The gap was most pronounced on long-tail targets (TL1A, B7-H3, NaPi2b, Claudin 18.2) with large preclinical and Asia-based developer pipelines
• All systems performed consistently on approved/Phase III anchor drugs (late-stage parity), confirming frontier LLMs retrieve high-visibility information effectively

Hallucination analysis:

• GPT + web and Gemini + web achieved perfect precision (1.000)
• Claude + web produced 1 hallucinated drug (precision 0.992); Perplexity produced 2 (precision 0.975)
• Error types included canonical-name collisions (different molecules sharing developer codes) and wrong-target attribution (e.g., rocatinlimab mislabeled as OX40L when it binds the OX40 receptor)

Latency: Gosset answered in sub-second time (a single database query); frontier systems were orders of magnitude slower (multiple web searches + generation):

\[\text{Latency Ratio} = \frac{T_{\text{frontier}}}{T_{\text{Gosset}}} \approx 10^2 \text{ to } 10^3\]

The Three-Layer Verification Pipeline

1. Deterministic Auto-Pass — drugs with industry-grade evidence (active trials, FDA approvals, sponsor commitments) auto-verified without LLM calls, clearing most uncontroversial cases. 2. Three-AI-Judge Cross-Check — remaining cases independently evaluated by Claude Opus 4.7, GPT 5.5, and Gemini 3.1 Pro (all with web search); a 2/3 majority produced an initial verdict (verified / hallucinated / unsure). 3. Human Expert Sign-Off — a pharma-pipeline reviewer adjudicated all edge cases, including canonical-name collisions, indirect pathway annotations, and wrong-target attributions. All reported numbers reflect post-review state.

> 📌 Note: Alias-aware union-find deduplication. To prevent systems inflating counts by listing aliases separately (e.g., "Tecotabart vedotin (LM-302, TPX-4589, BMS-986476)" counted as multiple entries), each system was required to return explicit alias lists, and an alias-aware union-find algorithm merged entries sharing any non-trivial alias.

Indexing Problem vs. Model Problem

The paper's core argument: the recall gap is an indexing problem, not a model quality problem.

Supporting evidence: 1. Reasoning was never in question — all frontier systems handled structured output, comprehension, and multi-step reasoning normally. 2. Late-stage parity — all systems correctly retrieved highly visible approved/Phase III drugs. 3. Long-tail blind spot — the gap concentrated in preclinical and early-clinical programs that exist on sponsor sites, conference posters, patents, and Chinese-language materials but are poorly indexed by search engines.

This has implications for MCP (Model Context Protocol) architecture: Gosset exposes its curated index as an MCP server, meaning frontier LLMs can switch information sources via tool calling without changing the model itself:

\[\text{Effective Recall} = f(\text{Model Reasoning}, \text{Index Coverage})\]

When index coverage has structural gaps, improvements in model reasoning cannot compensate for lost recall.

Practical Implications

1. Distinguish "processing length" from "recall coverage." "1M-token context" measures how much input a model can process, not how many facts it can recall. Measure effective recall separately for exhaustive-enumeration tasks. 2. Audit source coverage blind spots. Before using general LLMs for competitive intelligence, quantify how much of the target domain the open web covers and whether missing long-tail information (regional developers, academic projects, preprints) affects decision quality. 3. Consider curated indices as MCP tools. Wrapping a curated index as a callable tool—rather than replacing the LLM—preserves reasoning ability while fundamentally improving retrieval. The "model + curated index" hybrid may be optimal for specialized domains. 4. Build domain-specific verification pipelines. The three-layer architecture (auto-pass → AI cross-check → human expert) is a reusable template, especially for high-stakes decisions.

Limitations and Future Directions

Limitations: ① target selection may favor domains where Gosset's data is rich—for broadly covered targets (PD-1, HER2) the gap may be smaller; ② the AI judges are themselves LLMs with inherent calibration error; ③ absolute pipeline recall is unmeasurable—the cross-system union only reflects the discoverable universe, not the full real-world pipeline.

Future work: benchmarking frontier LLMs combined with the Gosset MCP server; extending the framework to other domains (legal precedent, patent landscaping, supply-chain intelligence); and automating curated index construction to lower production costs.

---

📚 Paper details

• Title: *Curated AI beats frontier LLMs at pharma asset discovery*
• arXiv ID: 2605.04908
• Published: 2026-05-06
• Authors: Łukasz Kidziński, Kevin Thomas
• Length: 5 pages, 5 figures, 1 table
• Key numbers: Gosset 451 verified drugs vs GPT+web 140 (3.2×); Gosset precision 1.000 / recall 100% of the 451-drug cross-system union; Claude+web 1 hallucination (P=0.992); Perplexity 2 hallucinations (P=0.975)
• Latency: Gosset sub-second (MongoDB round-trip); frontier systems roughly two orders of magnitude slower
• Core insight: the recall gap is an indexing problem, not a model problem—the open web does not catalog specialized long-tail content, leaving frontier LLM inputs structurally incomplete