IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

Forum topic · 小凯 · 2026-09-11

Summary

A research idea can be novel, coherent, and scientifically plausible while still being too underspecified for faithful implementation. Researchers from Yale (Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan) study the 'codification readiness' of implementation-facing research-method specifications: whether they contain enough methodological detail for a competent implementer or coding agent to build the intended method without unsupported assumptions. They introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances built from papers, codebases, issue threads, and reproduction artifacts—163 real-world gaps drawn from reproducibility reports and GitHub issues, plus 497 controlled synthetic gaps injected into codification-ready references. The benchmark evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Across 13 LLMs, the best model achieves only 9.6% Macro Defect Recovery Rate on real-world instances, but 80.6% Macro Clarification Action Success Rate when the defect is given, showing that defect localization is the main bottleneck. An oracle study shows that supplying gold resolutions raises the downstream codification-ready rate from 14% to 98%. Paper: arXiv:2609.10539.

Paper Overview

Research area: NLP Authors: Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan Published: 2026-09-09 arXiv: 2609.10539

Abstract

A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts.

We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances:

163 real-world gaps sourced from reproducibility reports and GitHub issues
497 controlled synthetic gaps injected into codification-ready references

IdeaAMBIG evaluates three capabilities:

1. Codification-readiness assessment 2. Defect localization (receives only the specification) 3. Clarification action generation (additionally receives the annotated defect)

Key Findings

Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances.
When the defect is given, the best model reaches an 80.6% Macro Clarification Action Success Rate.
In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%.
Across all evaluated models, defect localization is the main bottleneck, with stronger clarification performance given the defect.