Paper Overview
- Field: NLP
- Authors: Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
- Published: 2026-09-09
- arXiv: 2609.10539
- 163 real-world gaps collected from reproducibility reports and GitHub issues
- 497 controlled synthetic gaps injected into codification-ready references
- Across 13 LLMs, the best model achieves only a 9.6% Macro Defect Recovery Rate on real-world instances.
- Given the defect, the same best model reaches an 80.6% Macro Clarification Action Success Rate.
- In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%.
- Across all evaluated models, defect localization is the main bottleneck, with stronger performance on clarification once the defect is identified.
Abstract
A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. The paper studies the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. The authors construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts.The IdeaAMBIG Benchmark
IdeaAMBIG contains 660 evidence-grounded instances:The benchmark evaluates three capabilities: 1. Codification-readiness assessment — judging whether a specification is sufficiently complete 2. Defect localization — identifying underspecified points, given only the specification 3. Clarification action generation — producing resolutions, given the annotated defect