Frontier LLM Agents Approach Human Curator-Level Phenotype Ontology Curation

Forum topic · 小凯 · 2026-05-30

Summary

A new study by James P. Balhoff and Hilmar Lapp (arXiv:2605.28965) revisits the phenotype curation benchmark of Dahdul et al. (2018), which linked free-text phenotype descriptions from seven phylogenetic studies to ontology terms. Five frontier hosted LLMs from Anthropic and OpenAI were deployed as "agentic curators" in self-contained workspaces containing the source publication PDFs, original annotation guidelines, four reference ontologies (UBERON, PATO, BSPO, GO), and validation scripts. Evaluated against the same gold standard, every agent's annotations fell within the range of inter-curator variability of three trained human biological curators, with the best-performing agents approaching but not exceeding the top human curator. All agents substantially outperformed the Semantic CharaParser NLP tool on all four evaluation metrics. The results suggest that LLM-based agents can effectively alleviate the labor-intensive bottleneck of phenotype annotation for comparative morphology data integration.

Paper Overview

Research Area: Agents / Bioinformatics Authors: James P. Balhoff, Hilmar Lapp Published: 2026-05-30 arXiv: 2605.28965

Background

Linking free-text phenotype descriptions to ontology terms (phenotype annotation) is essential for cross-study integration of comparative morphological data. This labor-intensive process has long relied on highly trained human experts, making scalability a critical bottleneck.

Dahdul et al. (2018) established a gold standard of EQ annotations spanning seven phylogenetic studies, and used it to evaluate three human curators alongside the Semantic CharaParser NLP tool.

Method

This study revisits that benchmark using five frontier hosted LLMs (from Anthropic and OpenAI), each operating as an "agentic curator" within a self-contained workspace. Each agent was provided with:

The source publication PDFs
The original annotation guidelines
Four project ontologies: UBERON, PATO, BSPO, GO
Validation scripts

Results

Evaluated against the same gold standard:

Every agent fell within the range of inter-curator variability observed among the three trained human biological curators.
The best-performing agents approached but did not reach the performance of the best human curator.
Agents substantially outperformed Semantic CharaParser on all four evaluation metrics.

Abstract (Original)

> Linking free-text phenotype descriptions to ontology terms is essential for cross-study integration of comparative morphological data. This labor intensive process has heavily relied on human experts. Here we revisit the benchmark with five frontier hosted LLMs from Anthropic and OpenAI, each operating as an "agentic curator" within a self-contained workspace. Evaluated against the same Gold Standard, every agent fell within the range of inter-curator variability; the best performing agents approached but did not reach the best performing human curator. Agents substantially outperformed Semantic CharaParser on all four metrics.

--- *Auto-collected on 2026-05-30*