From Benchmark to Bench: AI Agents Enter Real-World Drug Discovery Labs

This is the final chapter of a three-part series about what happens when AI steps out of the "gym" (benchmarks) into the real world. Part 1 covered the physical world's walls (latency, lying sensors); Part 2 covered human society's walls (intent that shifts mid-conversation). This article covers the third door: institutions - specifically, the labs of three real pharmaceutical companies.

The Paper

• Title: *From Benchmark to Bench: Can Agents Survive Real-World Drug Discovery?*
• Authors: Pierre Llompart, Levent Guner, Helen Lai, Alessandro Tibo, Yijie Xu (AMD Silo AI, Stockholm/Helsinki; AstraZeneca, Cambridge/Gothenburg)
• arXiv: arXiv:2610.06411 [cs.AI], 2026-10-05

Why This Is Hard

Lead optimization - turning a hit compound with good activity but poor solubility/unknown toxicity into a viable candidate - is one of the most expensive phases of drug discovery. It runs on the DMTA loop (Design-Make-Test-Analyze); each cycle takes weeks, dozens of cycles take years, and every property fights every other property.

REINVENT 4 is a production-grade molecular generation framework that blindly optimizes whatever scoring function experts feed it. Most prior drug-agent evaluations only check tool execution and generic benchmark scores - testing a chef's knife skills without ever tasting the food. MAGI moves the exam into the wild: real projects, real objectives, real time cut-offs, replayed retrospectively.

MAGI: The Orchestration Layer

MAGI runs Claude Sonnet 4.6 as its reasoning backend and exposes 24 tools over MCP (Model Context Protocol) in six groups: service discovery, data retrieval, TTP management, scoring/prediction, optimization control, and result retrieval.

Key design elements:

• TTP (Therapeutic Target Profile): a versioned, schema-validated JSON document - not chat vibes - specifying scorer, optimization direction, target values, desirability transform, and weights per endpoint, plus pass/fail guardrails. A mapper compiles TTPs into REINVENT scoring configurations.
• Derringer-Suich weighted geometric mean merges multi-objective scores. Geometric mean is deliberately cruel: a molecule with one catastrophic endpoint (score 0) cannot average its way to "acceptable."
• No LLM hallucination allowed: any endpoint without data is marked "not evaluable" - never filled in by the model.
• Scoring stack: global ADMET ensemble (XGBoost, Chemprop-MTL, CheMeleon-MTL; 156,797 molecules, 186,120 sparse labels, 7 endpoints), Sanofi project-specific ligand models, Boltz-2 co-folding, PoseBusters conformational validity, plus an interaction analyzer - all behind a uniform REST contract (POST /score). Structural evidence is passed by artifact identity, so PoseBusters checks exactly the complex Boltz-2 produced.
• Paranoid auditability: every message, tool call, TTP version, and scoring config goes into JSONL transcripts, with scientific adaptations (evidence-based goal changes) logged separately from execution retries.

Two Routes, Nine Campaigns

Two design routes competed:

• LLM route: the reasoning backend directly nominates ~5 molecules per round - a focused local optimizer.
• REINVENT route: the agent writes/updates TTPs, launches and monitors optimization jobs, retrieves the generated pool, and shortlists - a broad space explorer.

Nine retrospective campaigns (starting compound + 4 measured analogs only; 5 rounds of ~5 candidates each):

• Four GSK BET BD2 campaigns (GSK-S1 to S4, published in *J. Med. Chem.*)
• Four Sanofi RetroDMTA projects: FXa, MMP8, PPARδ, RENIN
• One undisclosed AstraZeneca campaign (aggregate logD7.4 only)

Future compounds were held out on both the generation and scoring sides.

Key Findings

• Both routes produced valid molecules with opposite personalities. The LLM route is a sniper: fewer operations, comparable or higher primary-endpoint activity, tight to the starting chemotype. REINVENT is an army: higher novelty and diversity, exploring far from the starting scaffold - at 4-9x the observable operation count (and ~100,000 extra structures explored internally per campaign, four orders of magnitude more tool calls).
• The ruler decides success, not the generator. Three GSK series plus the AZ campaign hit goals. The two failures map exactly to scoring-model applicability-domain failures:
• GSK-S1: the kinetic solubility target fell outside training-data coverage (extrapolation).
• RENIN: ranking correlation of only ~0.31 on held-out compounds (weak ordering). Both routes failed together - the referee's eyesight set the ceiling.
• Three scoring-model pathologies are identified: systematic bias, value-range compression, and weak rank correlation. No generator width can compensate.
• Sanofi campaigns: both routes matched the predicted primary activity of historical compounds, but only FXA and PPARδ crossed project thresholds - reproduced, not surpassed. The LLM route hit goals more often.

Highlight: The Agent Changed Its Own Ruler

In GSK-S1, facing a persistently unreachable kinetic solubility target, the agent autonomously promoted thermodynamic solubility as a proxy endpoint (physically related, and within model competence) - after which proposals showed real improvement (paper Figure S12). This is an *evidence-motivated* goal change, logged separately from error retries. The framework's real value lies not in calling 24 tools but in two harder judgments: which tool to choose, and when to rewrite the goal.

Lab Turing Test

Four chemists, 28 review questions: distinguish agent-designed molecules from held-out, historically human-made, experimentally tested compounds. They could not. No preference between design routes either. Their verdict on agent SAR reasoning: broadly credible but incomplete - clear thinking, shallow experience. Note: indistinguishable does not mean flawless - it means as good as the humans, with the same vulnerability to scoring-model myopia.

Structural Evidence Shakes Up the Shortlist

Boltz-2 co-folding provides 3D binding poses and affinity signals. Re-ranking top-5 shortlists by Boltz-2 affinity substantially changed the lists, target-dependently, with PoseBusters vetoing physically implausible poses. Structural evidence can actively steer optimization rather than just hold up a score sign at the finish line.

Conclusion: The Wall Is Not Inside the Model

Across the trilogy - physical world, human society, industrial institutions - the recurring theme: the real walls AI faces outside the gym are not inside the model. The scoring service is the oracle; the oracle's applicability domain is the agent's capability boundary. MAGI's methodology: treat scoring-service applicability as a first-class citizen - explicit TTPs, centralized scoring, identity-tagged evidence, logged changes. Future drug-agent competition will hinge not on wilder generators but on knowing precisely where your ruler fails.

Honest Limitations

• Proposed endpoints are predicted, not measured.
• One backend, one trajectory per condition; no static-target control group isolating the agent's own contribution.
• Temporal splits block explicit leakage but cannot guarantee pretrained models never saw the data.

References

• Llompart, P., Guner, L., Lai, H., Tibo, A., Xu, Y. *From Benchmark to Bench: Can Agents Survive Real-World Drug Discovery?* arXiv:2610.06411 [cs.AI], 2026-10-05.
• REINVENT 4: production-grade molecular generation framework (MAGI's optimization backend).
• Derringer, G., Suich, R. *Simultaneous Optimization of Several Response Variables.* Journal of Quality Technology, 1980.
• Boltz-2: protein-ligand co-folding and binding affinity prediction.
• PoseBusters: physical validity checks for docking poses.
• GSK BET BD2 chemical series (published in *J. Med. Chem.*), source of campaigns GSK-S1 to S4.
• Sanofi RetroDMTA projects: FXa, MMP8, PPARδ, RENIN.

*The three-part "AI Steps Outside" series is complete: physical world, human society, industrial institutions - three walls touched. The door is half open; the other half depends on giving AI a ruler that stays accurate in the wild.*