Paper Overview
Field: Machine Learning Authors: Yukun Dong, Linjun Zhang, Tian Zheng Published: 2026-10-19 arXiv: 2610.22465
Summary
Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. This paper presents an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment.
Key components:
- Structured checklist: Records reproduction instructions and limitations declared by the authors.
- Two-stage GitHub workflow: Connects agent checks to author clarification, revision, and agent reassessment.
- Separate agent assessments: The agent independently evaluates (1) whether results have been reproduced and (2) whether authors need to provide clarification or make repairs.
- Human oversight: Human reviewers approve requests and decide whether concerns are resolved.
Deployment and Results
The system was deployed across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all of the issues, although full reproduction remained unverified in some cases.
The authors also compared repeated audits conducted with complete versus blank checklists to examine how author declarations shaped the agent's investigation and assessment.
Contribution
The work contributes a collaborative workflow and agent design that leverage author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that collaboration among authors, agents, and reviewers can help identify and address reproducibility problems.
Original Abstract
> Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that this approach can help identify and address reproducibility problems through collaboration among authors, agents, and reviewers.
--- *Auto-collected on 2026-10-19*