Paper Overview
Field: Machine Learning Authors: Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi Published: 2026-03-26 arXiv: 2603.23562
Summary
This research explores a frontier problem in machine learning: how to help language models effectively acquire new parametric knowledge in data-constrained domains.
The authors note that synthetic data augmentation helps language models learn new knowledge in data-constrained domains. However, naively scaling existing synthetic data methods—by training on more synthetic tokens or using stronger generators—yields diminishing returns, with performance remaining below that of retrieval-augmented generation (RAG).
To break this "RAG ceiling", the paper introduces Synthetic Mixed Training, a method that combines synthetic QAs (question-answer pairs) and synthetic documents during training.
Original Abstract
> Synthetic data augmentation helps language models learn new knowledge in data-constrained domains. However, naively scaling existing synthetic data methods by training on more synthetic tokens or using stronger generators yields diminishing returns below the performance of RAG. To break the RAG ceiling, we introduce Synthetic Mixed Training, which combines synthetic QAs and synthetic documents.
Links
- Paper: https://arxiv.org/abs/2603.23562