Ideas Have Genomes: IG-Bench Shows AI Scientists Struggle to Trace Scientific Lineage

The Problem: Topical Similarity vs. Mechanism Inheritance

At a hypothetical "academic family reunion," papers like YOLO, YOLOv2, DETR, Faster R-CNN, CLIP, LLaVA, BERT, Transformer, and ViT all look related—they work on object detection or vision-language tasks. But looking similar isn't being related. YOLOv2 genuinely inherits YOLO's single-stage detection (with fixes: anchor boxes, batch normalization, multi-scale training). DETR competes in the same niche but replaced Faster R-CNN's CNN region proposals with Transformer set prediction—same village, different lineage.

In July 2026, a team led by Xue Yang (Shanghai Jiao Tong University) with Tsinghua, CMU, and others released IdeaGene-Bench (IG-Bench), applying an evolutionary-biology framework to test 14 LLM scientists. The result: even the strongest system (Claude Code + GPT-5.5) scored only 27.3%.

The core insight: scientific progress is mechanism inheritance, not topical proximity. Papers can share a task without inheriting from each other (competition), or be textually distant while carrying the same core mechanism (adaptive radiation).

Idea Genome: The Unit of Inheritance

Each paper is decomposed into Idea Genome objects, each with:

• Role type: niche, mechanism, observation, limitation, delta, or claim
• Content description
• Evidence pointer to specific passages, figures, or equations
• Optional constraints

Four key constraints: typed (not unstructured summaries), evidenced (no free inference), minimally self-contained (independently inheritable/mutable/lost), and lineage-relevant.

GenomeDiff and Six Evolutionary Dynamics

Given a predecessor \(p_s\) and successor \(p_t\), GenomeDiff \(\Delta_{s \to t}\) records each genome object's fate: Inherited, Mutated, Lost, Novel, or External. Diff patterns map to six dynamics:

| Dynamic | Lineage? | Criterion | Example | |---------|----------|-----------|---------| | Mutation | Yes | Mechanism inherited with local fixes, niche unchanged | YOLO → YOLOv2 | | Adaptive Radiation | Yes | Mechanism retained, new task/domain | Transformer → ViT | | Hybridization | Yes | Driver imported from multiple lineages | CLIP + instruction-tuned LLM → LLaVA | | Speciation | Yes | Same niche, driver mechanism replaced | Faster R-CNN → DETR | | Niche Competition | No | Same niche, no driver inheritance | Faster R-CNN vs. YOLO | | Isolation | No | Neither shared niche nor inheritance | BERT vs. YOLO |

Ambiguities resolve by fixed priority: hybridization > speciation > niche competition; adaptive radiation > mutation. This is an auditable evaluation contract, not a complete theory of science history.

IG-Bench: 1,961 Gold Lineages

• 1,961 gold lineage trajectories across 10 fields (NLP, vision, multimodal, biology, chemistry, physics, materials, medicine, math)
• 1,085 curated Idea Genome objects, 920 pairwise GenomeDiff records
• Trajectories contain 3–7 papers via citations, semantic search, and curation

Evaluation has two parts:

IG-Exam (closed-book): 42 task types, 1,029 instances, four capability tiers—T1 genome abstraction, T2 inheritance tracking, T3 evolutionary inference, T4 lineage verification.

IG-Arena (open generation): proposals written under Question-only / Library / Lineage settings, scored by Population-Evolution Score (PES): Heredity, Variation, Selection.

Findings from 14 AI Scientists

Systems tested: direct LLMs (GPT-5.5, Claude Opus 4.7, Qwen3.6, Gemini-3.1, Kimi-K2, DeepSeek-V4, GLM-5.1, MiniMax-M2.7), research agents (AI Scientist v2, CoI-Agent), CLI toolchains (Codex, Claude Code).

1. Lineage reasoning is a compositional challenge

Best IG-Exam score: 27.3% (Claude Code + GPT-5.5); best direct LLM: 23.1% (GPT-5.5).

| Capability | Best score | |-----------|------------| | T1 Genome abstraction | 34.4% (Gemini-3.1) | | T2 Inheritance tracking | 37.9% (Claude Code + GPT-5.5) | | T3 Evolutionary inference | 25.3% (Claude Code + GPT-5.5) | | T4 Lineage verification | 17.4% (Qwen3.6) |

Errors are rarely simple retrieval failures—models find the right parent paper but mislabel the dynamic, or identify the driver genome but misjudge its fate. This is a consistency failure, not an information gap.

2. Tool scaffolds help retrieval, not reasoning

CLI toolchains substantially boost T2 (GPT-5.5: 25.7% → 37.9%) via iterative cross-paper retrieval, but gains shrink at T3 and nearly vanish at T4. Research agents overlap with direct GPT-5.5 on most dimensions and even score worse on generation (PES). Scaffolds amplify retrieval-dependent abilities; the compositional bottleneck is unmoved.

3. Plausibility > lineage coherence

PES decomposition shows:

• Variation nearly constant (82.7–84.7) — no lack of novelty
• Selection nearly constant (79.7–82.2) — ideas "sound useful"
• Heredity is the only differentiator (61.9 → 84.2)

Question-only Mutation mode scores Heredity 61.9 (PES 69.2); Lineage-setting Hybridization reaches 84.2 (PES 83.6). The gain from Question to Lineage is not more innovation—it's better grounding. Systems overproduce plausibility rather than lineage coherence.

Implications for AI Research

1. Retrieval ≠ understanding. Retrieval finds co-occurring papers but not inherited mechanisms; genome-level alignment is needed to distinguish competition from adaptive radiation. 2. Compositional consistency is the real bottleneck. T4's 17.4% reflects multi-constraint verification, consistent with prior "Knowing-Using Gap" findings. Auto-research needs compositional verification modules, not just better retrieval. 3. "Sounds right" vs. "is right." Generated ideas can be high-variation and low-heredity—novel-sounding but disconnected from the correct parent mechanism, a fatal flaw in peer review. 4. Evolutionary biology is a rich source for AI evaluation. Darwin's "descent with modification," 167 years later, becomes an operational, auditable framework for judging whether AI understands how scientific ideas evolve.

Commentary: The Granularity Problem

The author's main open question is Idea Genome granularity. The paper's criterion—split components that can independently disappear or transfer; merge those unintelligible apart—is a functional-dependency test that is hard to apply. Is multi-head attention one genome, or are "QK matrix multiplication" and "head splitting" separate? Biology faced the same issue: the definition of "gene" evolved from Mendel's factors to coding sequences to modern regulatory-plus-coding constructs. Idea Genome likely needs similar refinement.

Still, IG-Bench has done something important: it turned "can AI understand the inheritance structure of scientific ideas?" from a philosophical question into a measurable one. At 27.3%, we're far from a true AI scientist—but at least we now know what to measure.

---

Paper: arxiv.org/abs/2607.08758

Code and data: github.com/VisionXLab/IdeasHaveGenomes

Project page: visionxlab.github.io/IdeasHaveGenomes