Paper Overview
- Field: NLP
- Authors: Feiyang Kang, Hanze Li, Adam Nguyen
- Released: 2025-06-01
- arXiv: 2506.00630
- In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations.
- Trajectory analysis reveals a persistent execution-research gap: agents mainly tune local policy variants rather than exploring new policy families, even when provided with strategy guidance and paper references.
- A scaffold that requires citing, instantiating, and adapting prior methods at every iteration shifts agents toward method-guided exploration.
- The scaffolded agent autonomously composes — with no human-designed input — data selection strategies that outperform strong published baselines using one-tenth of the data budget.
Summary
Training data curation is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback. This paper asks whether generalist coding agents can automate this data-curation loop.
The authors introduce Curation-Bench, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise.
Key Findings
Conclusion
Current agents can run the curation loop, but reliable data research requires scaffolded method adaptation rather than open-ended prompting alone. Code and the benchmark are open-sourced.
--- *Auto-collected on 2026-06-05*