[论文] Author-in-the-loop Agentic Reproducibility Auditing for a Machine L...

论文概要

研究领域: ML
作者: Yukun Dong, Linjun Zhang, Tian Zheng
发布时间: 2026-10-19
arXiv: 2610.22465

中文摘要

同行评审中的可复现性核查，需要关于作者代码「应复现什么」的共识。本文提出作者在场（author-in-the-loop）的协作审计：结构化清单记录复现说明与局限；GitHub 上的两阶段工作流把智能体检查与作者澄清、修订、复评连接起来。智能体分别评估「结果是否被复现」与「作者是否需要澄清或修复」；人类审稿人批准请求并决定关切是否解决。该系统在 36 篇会议论文、133 个实验表格与图上部署，经人工复核提出 20 个问题；进一步核查支持关闭全部问题，尽管部分案例未完全验证复现。研究还比较了完整清单与空白清单下的重复审计，考察作者声明如何塑造智能体的调查与评估。工作贡献了一个利用作者知识引导可复现性审计的协作工作流与智能体设计；会议部署提供了实证证据：作者、智能体、审稿人三方协作能帮助识别并解决可复现性问题。

原文摘要

Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that this approach can help identify and address reproducibility problems through collaboration among authors, agents, and reviewers.

自动采集于 2026-10-19

#论文 #arXiv #ML #小凯