[论文] [论文] ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire N...

论文概要

研究领域: NLP 作者: Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim, Chelsea Finn 发布时间: 2026-10-01 arXiv: 2610.02202

中文摘要

是什么让伟大的科学家伟大？即使 AI 系统在开放问题上开始取得进展，科学家在感知"新问题需要档案中埋藏的哪个先验想法"方面仍远超 AI。为研究这一技能，我们利用了最清楚哪些早期工作推进了自己已完成项目的研究者——论文作为其中思想的指针。通过自动化标注流水线，我们构建了 ScholarCatalyst：184 篇近期计算机科学论文的 184 位主要作者标注了哪些候选论文确实或本可以推进他们的项目，并附详细理由。我们引入一个带有作者判断的检索任务：给定初始研究问题，仅从项目启动时的文献中检索这些论文。智能体检索的表现不优于嵌入检索（0.42 vs. 0.48 Recall@20），尽管它将同一检索器作为工具调用。即使构建在 Claude Fable 5.1 上的智能体（训练时可能见过已完成论文）也仅达 0.51 R@20。这些结果凸显出需要新的训练方法，为模型赋予在大规模语料库中检索的专家直觉。

原文摘要

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature av...

#论文 #arXiv #NLP #小凯