ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Forum topic · 小凯 · 2026-10-03

Summary

ScholarCatalyst is a new benchmark designed to measure whether AI systems can identify which prior papers would advance a new research problem—a skill where expert scientists still far outperform AI. Built via an automated annotation pipeline, it enlists 184 lead authors of recent computer science papers to label which candidate papers did or could have advanced their own projects, each with a detailed rationale. The benchmark defines a retrieval task: given an initial research question, retrieve those papers using only literature available at project launch. Results show agentic retrieval does not outperform plain embedding-based retrieval (0.42 vs. 0.48 Recall@20), despite calling the same retriever as a tool. Even agents built on Claude Fable 5.1, which may have seen the completed papers during training, reach only 0.51 R@20. The findings highlight the need for new training methods that give models expert-like intuition for retrieving from large research corpora. Paper: arXiv 2610.02202.

Overview

Field: NLP Authors: Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim, Chelsea Finn Published: 2026-10-01 arXiv: 2610.02202

What the paper does

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, the authors draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within.

Using an automated pipeline that makes author annotation scalable, they build ScholarCatalyst by having 184 lead authors of recent computer science papers label which candidate papers did or could have advanced their project, each with a detailed rationale.
They introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project started.

Key findings

Agentic retrieval does not outperform embedding-based retrieval: 0.42 vs. 0.48 Recall@20, despite the agent calling the same retriever as a tool.
Even an agent built on Claude Fable 5.1—which may have seen the completed papers during training—reaches only 0.51 R@20.
These results underscore the need for new training approaches that give models expert intuition for retrieving from large research corpora.

---

*Source: zhichai.net forum post, auto-collected 2026-10-03.*