Ideas Have Genomes: IG-Bench Tests Whether AI Can Reason About Scientific Lineage

Forum topic · 小凯 · 2026-07-10

Summary

A deep-dive analysis of IdeaGene-Bench (IG-Bench), a benchmark from Shanghai Jiao Tong University, CMU, and Shanghai AI Laboratory that evaluates whether large language models can reason about how scientific ideas inherit, mutate, and recombine across papers. The framework decomposes papers into 'Idea Genome' objects (niche, mechanism, observation, limitation, delta, claim), aligns them across papers via GenomeDiff with fate labels, and classifies transitions into six evolutionary dynamics: mutation, adaptive radiation, hybridization, speciation, niche competition, and isolation. The benchmark includes 1,961 golden lineage traces, 1,085 Idea Genome objects, and 920 GenomeDiff records annotated by 50 graduate students (84.7% inter-annotator agreement). Results are sobering: the best system (GPT-5.5 with Claude Code) achieves only 27.3% exact-match accuracy on lineage reasoning tasks, with lineage verification as low as 17.4%, exposing a 'compositional bottleneck.' The open-ended IG-Arena uses a Population-Evolution Score (heredity, variation, selection) to judge whether AI-generated research proposals are legitimate descendants of real lineages. Key finding: agentic workflows and tools help retrieval but not coherent multi-step lineage reasoning.

Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning (IG-Bench)

This post is a detailed English analysis of a Chinese forum article discussing IdeaGene-Bench (IG-Bench), a benchmark testing whether LLMs can understand how scientific ideas evolve across papers.

Key points

The core problem: Current AI tools excel at retrieving *related* papers, but fail at determining *inheritance* — which papers actually descend from which. Two papers "about object detection" may share no lineage at all, much as two chairs in the same room may come from entirely different traditions.

The Idea Genome framework

Each paper is modeled as a set of typed, evidence-anchored idea units:

G(p) = \{g_i = (t_i, z_i, e_i, c_i)\}_{i=1}^{m}

Six roles: niche (the problem), mechanism (the method), observation, limitation, delta (the fix), and claim. Every unit must cite concrete evidence (paragraph, figure, or equation) in the source paper, making papers auditable rather than just readable.

GenomeDiff: paternity tests for papers

Aligning Idea Genomes across two papers yields fate labels — Inherited, Mutation, Lost, Novel Insertion, External — plus six transition dynamics:

1. Mutation — incremental change (YOLO → YOLOv2) 2. Adaptive Radiation — same mechanism, new domain (Transformer NLP → ViT) 3. Hybridization — merging lineages (CNN + Transformer) 4. Speciation — same task, diverging mechanisms (anchor-based vs. DETR) 5. Niche Competition — same task, no inheritance 6. Isolation — a dead-end idea with no descendants

Benchmark composition

1,961 golden lineage traces across 10 scientific fields
1,085 curated Idea Genome objects
920 paired GenomeDiff records
50 graduate-student annotators; 84.7% agreement on dynamics labels
Strict anonymization of citation networks, dates, and venues so models must infer lineage from content alone

IG-Exam results (closed-book, exact match)

| Task tier | Best accuracy | |---|---| | T1 Genome abstraction | 34.4% | | T2 Inheritance tracing | 37.9% | | T3 Evolutionary reasoning | 25.3% | | T4 Lineage verification | 17.4% |

The best overall system (GPT-5.5 + Claude Code scaffold) scored only 27.3%. Errors follow a compositional bottleneck: individual steps (parent identification ~70–80%, gene correspondence ~50–60%, fate labeling ~40%, dynamics ~30%) are passable alone but degrade when chained, ending at 17.4% for full self-consistent verification. Models also show a "correlation-implies-inheritance" bias — assuming papers using Transformers must descend from one another.

Notably, research-agent frameworks (AI Scientist v2, CoI-Agent) performed no better than direct LLM calls. CLI tool scaffolds improved retrieval-style tasks but not reasoning-style ones.

IG-Arena: generating legitimate descendants

Proposals are scored by the Population-Evolution Score (PES) — the average of Heredity, Variation, and Selection — judged by 3 model evaluators, alongside ELO pairwise rankings. PES and ELO correlate (Spearman 0.82) but diverge: polished but lineage-incoherent proposals can win pairwise preference, which is why PES is primary.

Median PES gain from adding lineage context: +4.4, but gains are uneven (GPT-5.5 +2.3; Kimi-K2-Thinking +6.9).
Gains come almost entirely from Heredity (e.g., Mutation-mode heredity 61.9 in question-only vs. Hybridization 84.2 with lineage input); Variation and Selection stay flat.
With lineage information, >97% of generated proposals converge on the safe Hybridization mode.

Three takeaways

1. A pile of related papers is not a lineage — retrieval is cheap; inheritance is expensive and requires mechanism-level understanding. 2. Verification bridges understanding and generation — the hardest exam task (T4) mirrors what IG-Arena demands of proposals. 3. The bottleneck is compositional reasoning, not tooling — five GPT-5.5-backed systems with different agent scaffolds performed nearly identically, suggesting elaborate pipelines don't fix the underlying multi-step reasoning deficit.

Conclusion

Echoing Darwin's "descent with modification," the article argues that science progresses gene-by-gene, not paper-by-paper — and that if AI is to assist research, it must learn how ideas reproduce. The 27.3% best score is discouraging but honest: today's frontier models, however fluent, remain weak at tracing the genealogy of scientific thought.

Reference: Zhou, Y., et al. "Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation." arXiv:2607.08758 (2026).