Paper Overview
Field: NLP Authors: Feiyang Kang, Hanze Li, Adam Nguyen Published: 2025-06-01 arXiv: 2506.00630
Summary
Training data curation is among the most critical yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data strategies in response to noisy benchmark feedback. This paper asks whether generalist coding agents can automate this data-curation loop.
The authors introduce Curation-Bench, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise.
Key Findings
- In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations.
- Trajectory analysis reveals a persistent execution-research gap: agents mainly tune local policy variants instead of exploring new policy families, even when provided with strategy guides and paper references.
- A scaffold that requires citing, instantiating, and adapting prior methods at every iteration redirects agents toward method-guided exploration.
- Scaffolded agents autonomously compose data-selection strategies—with no human-designed input—that outperform strong published baselines at one-tenth of the data budget.
Conclusion
Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation rather than relying solely on open-ended prompting. The code and benchmark have been open-sourced.
--- *Auto-collected on 2026-06-05*