IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

Forum topic · 小凯 · 2026-09-11

Summary

A research idea can be novel and scientifically plausible yet still too underspecified for faithful implementation. The IdeaAMBIG benchmark (arXiv:2609.10539) studies the codification readiness of implementation-facing research-method specifications: whether they contain enough methodological detail for a competent implementer or coding agent to build the intended method without unsupported assumptions. The benchmark comprises 660 evidence-grounded instances: 163 real-world gaps drawn from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready reference specifications. It evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Across 13 evaluated LLMs, the best model achieves only 9.6% Macro Defect Recovery Rate on real-world instances, while reaching 80.6% Macro Clarification Action Success Rate when the defect is given. An oracle study shows that supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. These results identify defect localization as the primary bottleneck for LLMs in translating research ideas into faithful implementations.

Paper Overview

Field: NLP

Authors: Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan

Published: 2026-09-09

arXiv: 2609.10539

Abstract

A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. This work studies the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. The authors construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts.

The IdeaAMBIG Benchmark

IdeaAMBIG contains 660 evidence-grounded instances:

163 real-world gaps sourced from reproducibility reports and GitHub issues
497 controlled synthetic gaps injected into codification-ready reference specifications

The benchmark evaluates three capabilities:

1. Codification-readiness assessment 2. Defect localization — the model receives only the specification 3. Clarification action generation — the model additionally receives the annotated defect

Key Findings

Across 13 LLMs, the best model achieves only 9.6% Macro Defect Recovery Rate on real-world instances.
When given the defect, the same best model reaches 80.6% Macro Clarification Action Success Rate.
In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%.
Across all evaluated models, defect localization is the main bottleneck, with substantially stronger clarification performance given the defect.

Source

Full paper: arXiv:2609.10539