SynPro: Using Reinforcement Learning to Multiply Effective Pretraining Data via Rewriting

Forum topic · 小凯 · 2026-05-19

Summary

Large language model pretraining is shifting from compute-bound to data-bound scaling. SynPro, a framework by Yu and Xiong (CMU), uses reinforcement learning to optimize two operations—rewriting and reformatting—to generate diverse training tokens from the same organic data. Rewards are driven by three dimensions: quality, faithfulness, and data influence. In data-constrained settings (using only 10% of Chinchilla-optimal tokens), SynPro unlocks 3.7–5.2x more effective tokens than standard repetition, and even surpasses a non-data-constrained control group at the 1.1B parameter scale, without causing distribution collapse. Open questions include whether rewriting and reformatting introduce repetitive patterns or artifacts into the data, how the RL optimizer handles the subjectivity of quality rewards, and how SynPro-generated data performs when mixed with other data sources such as human-written text.

LLM pretraining is shifting from compute-bound to data-bound scaling. SynPro, a framework proposed by Yu and Xiong (CMU), uses reinforcement learning to optimize two operations—rewriting and reformatting—to generate diverse training tokens from a single copy of organic data.

Key points

Reward design: Rewards are driven by three dimensions — quality, faithfulness, and data influence.
Data-constrained results: With only 10% of Chinchilla-optimal tokens, SynPro unlocks 3.7–5.2x more effective tokens than standard repetition.
Beyond the baseline: At the 1.1B parameter scale, SynPro even outperforms a "non-data-constrained" control group.
No distribution collapse: The framework does not lead to distribution collapse.

Open questions

Do rewriting and reformatting introduce repetitive patterns or artifacts into the data?
The RL optimizer depends on quality rewards — who judges what counts as "high quality" given their subjectivity?
How effective is SynPro-generated data when mixed with other data sources, such as human-written text?

References

1. Yu, Z., & Xiong, C. (2026). *Generating Pretraining Tokens from Organic Data for Data-Bound Scaling*. arXiv:2605.17849 [cs.CL]. 2. Hoffmann, J., et al. (2022). *Training Compute-Optimal Large Language Models*. NeurIPS. 3. Muennighoff, N., et al. (2024). *Scaling Data-Constrained Language Models*. NeurIPS.