[论文] IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications (arXiv:2609.10539)

论文概要

研究领域: NLP 作者: Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan 发布时间: 2026-09-09 arXiv: 2609.10539

中文摘要

一个研究想法可能新颖、连贯且科学上合理，但其提出的方法可能仍未被充分指定以支持忠实实现。本文研究面向实现的研究方法规范的「编码就绪性」(codification readiness)，定义为是否提供了足够的方法论信息，使有能力的实现者或编码智能体能够在无无支持假设的情况下构建预期方法。作者从论文、代码库、issue线程和复现构件中构建基于证据的规范及其支持性解决方案。本文提出 IdeaAMBIG 基准，包含660个基于证据的实例：163个来自可复现性报告和GitHub issue的真实世界差距，以及497个注入到编码就绪参考中的受控合成差距。该基准评估三种能力：编码就绪性评估、缺陷定位和澄清行动生成。在13个LLM中，最佳模型在真实实例上的宏观缺陷恢复率仅为9.6%，但在给定缺陷时澄清行动成功率达80.6%。缺陷定位是所有模型的主要瓶颈。

原文摘要

A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.

---

*自动采集于 2026-09-11*

#论文 #arXiv #NLP #小凯