[论文] ReproBridge: Building Reproducibility into ML Research Artifacts

论文概要

研究领域: AI
作者: Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister, Jinsung Yoon
发布时间: 2026-11-05
arXiv: 2611.08452

中文摘要

复现机器学习实验需要从散落在论文、发布的代码和辅助制品中的信息重建流程。对编程 Agent 来说，复现成功与否往往取决于它们填补缺失细节的能力，而不仅仅是发布的制品本身。我们提出问题：保存已恢复的流程和决策，能否改善后续独立复现？我们定义了一种标准化制品设计，包含精炼代码库、环境配置、实验启动器和复现指南，并提出 ReproBridge——一个构建这些精炼可复现制品的 Agent 工作流。ReproBridge-Bench 提供来自 36 篇论文 387 个实验的 1,029 个任务，每个任务锚定五个经人工验证的参考分数。Agent 在全新会话中分别使用原始代码和精炼制品尝试任务；独立的审计员检查执行证据。在四个被评模型上，精炼制品将平均复现成功率从 76.4% 提升至 92.6%，平均降低下游 API 开销 19.7%。跨模型成功率差距从 21.6 个百分点缩小至 3.9 个百分点，所有四个模型的平均成功率均超过 90%。消融研究表明移除任何组件都会降低成功率，其中代码和环境移除的损失最大。这些结果表明，保存执行知识能让即使是原始代码成功率较低的模型也能更可靠、更低成本地复现实验。

原文摘要

Reproducing machine learning experiments requires reconstructing procedures from information scattered across papers, released code, and supporting artifacts. For coding agents, reproduction success can therefore depend on their ability to fill in missing details, rather than on the released artifacts alone. We ask whether preserving recovered procedures and decisions improves subsequent independent reproduction. We define a standardized artifact design comprising a refined codebase, environment setup, experiment launchers, and a reproduction guide, and introduce ReproBridge, an agent workflow for constructing these refined reproducible artifacts. ReproBridge-Bench provides 1,029 tasks from 387 experiments across 36 papers, anchored by five human-verified reference scores per task. Agents attempt tasks in fresh sessions with raw code or refined artifacts; a separate Auditor checks execution evidence. Across four evaluated models, refined artifacts raise mean reproduction success from 76.4% to 92.6% and reduce downstream API spending by 19.7% on average. The cross-model success-rate spread narrows from 21.6 to 3.9 percentage points, with all four models achieving mean success rates above 90%. Removing any component reduces success in the ablation study, with the largest losses from code and environment removal. These results suggest that preserving execution knowledge helps even models with lower raw-code success reproduce experiments more reliably and at lower API cost.

自动采集于 2026-11-08

#论文 #arXiv #AI #小凯