MGA Data Augmentation: Multimodal Technique for LLM Training Data

Forum topic · 小凯 · 2026-03-27

Summary

MGA (Multimodal Data Augmentation) is a lightweight framework that restructures existing corpora into diverse variants to address data scarcity and repetition degradation in large language model training. It leverages a 3.3B-parameter Mixture of Experts (MoE) model to automatically extract five Genre-Audience pairs from each source document—such as academic paper for researchers or dialogue for children—then uses a lightweight post-trained model to rewrite content according to each pair's style, depth, and structure while preserving core information. Quality is enforced by an LLM judge using a limited-consistency criterion that filters outputs below a score of 3. Experimental results show MGA achieves roughly 3.9x token expansion, reduces validation loss under repeated training cycles, and improves accuracy across reasoning benchmarks at multiple model scales, supporting efficient training of billion-parameter models.

MGA Data Augmentation

> Multimodal data augmentation technique for large language model training

Overview

MGA (Massive Genre-Audience augmentation) is a lightweight framework that restructures existing corpora into diverse variants, tackling two persistent challenges in LLM training:

Data scarcity. High-quality corpora are limited; after strict filtering, public datasets retain less than 10% of their original content.
Repetition degradation. Over-repeated training cycles degrade model generalization and optimization stability.

The core idea: generate semantically rich new data that preserves core knowledge while expanding training tokens by roughly 3.9x.

Core Concepts

Massive (Scale)

Each inference generates 5 Genre–Audience pairs.
One source document expands into 5 new documents.
Token count grows ~3.9x; efficient training of multi-billion-parameter models is supported.
One popular-science article → 5 different expression forms.
Coverage ranges from beginner to expert; suitable for large-scale corpus expansion.

Genre

Purpose: education, analysis, narration, etc.
Structure: step-by-step tutorial, academic paper, dialogue.
Style: rigorous academic vs. plain storytelling.
Depth: beginner intro vs. expert deep dive.

Examples include academic paper ↔ children's story, technical doc ↔ conversational tutorial, news report ↔ analytical report.

Audience

Demographics: age, occupation, education.
Background & motivation: expertise level, learning goals.
Use case: work needs, hobby, academic research.

Examples: middle-school student vs. medical graduate researcher; office worker vs. med student; programming beginner vs. senior engineer.

Technical Pipeline

The MGA pipeline has three stages:

1. Genre–Audience Pair Generation

A 3.3B-parameter MoE model adaptively extracts Genre–Audience combinations from the source document.
It analyzes the original text and automatically produces 5 distinct pairs (e.g., "academic paper–researcher", "dialogue–elderly person").
Each pair defines an expression framework and audience characteristics.

2. Text Reconstruction

A lightweight, post-trained tool model rewrites the text according to each Genre–Audience requirement.
It adjusts structure and language style to match the genre.
It calibrates knowledge depth and expression to fit the audience.
Core information must remain complete and accurate.

Example transformations:

*Dialogue for primary-school children:* simplified terms + concrete examples + Q&A format.
*Academic paper for researchers:* IPCC AR6 data, CO₂ concentration 415 ppm, ~1.1 °C warming since 1850.

3. Quality Control

An LLM judge model performs limited-consistency evaluation.
Style and ordering may vary, but core information must be traceable to the source.
Outputs scoring below 3 are filtered out.

Limited-consistency criteria:

✓ Core information points preserved
✗ Excessive semantic deviation
Score ≥ 3 required

Interactive Demo (Climate Change Sample)

Source text (excerpt): > Climate change refers to long-term shifts in Earth's climate system. Main causes include rising greenhouse-gas emissions, which drive up global average temperature. This causes polar ice melt, sea-level rise, and more frequent extreme weather. Scientists study this with observational data and climate models, and propose response strategies.

MGA generates:

| Genre | Audience | |---|---| | Academic paper | Climate-science researcher | | Children's story | Ages 6–10 | | News report | General public | | Policy document | Government decision-maker | | Teaching material | High-school student |

Sample outputs:

Academic paper–researcher: *Based on IPCC AR6 data, global mean temperature has risen ~1.1 °C since 1850; CO₂ concentration has reached a historical high of 415 ppm, closely linked to human activity.*
Children's story–ages 6–10: *Once, Mother Earth wore a beautiful atmosphere coat. But recently, the kids noticed she was getting hotter. It turns out smoke from factories and cars made the coat thicker, trapping the heat…*
News report–general public: *According to the latest scientific report, global temperatures continue to rise. Experts say this is closely linked to greenhouse-gas emissions from industrial activity, affecting daily life through more frequent extreme weather…*
Policy document–government decision-maker: *Under the Paris Agreement goals, nations must set concrete carbon-reduction plans: (1) advance clean-energy transition; (2) strengthen forest protection and afforestation; (3) implement carbon pricing; (4) promote international cooperation.*
Teaching material–high-school student: *Key points: (1) Greenhouse effect mechanism; (2) main GHGs: CO₂, CH₄, N₂O; (3) observational evidence; (4) impact assessment on ecosystems, agriculture, and society.*

Results

Token expansion: ~3.9x growth from baseline data.
Reasoning accuracy: consistent gains across multiple reasoning benchmarks.
Cross-scale performance: improvements hold across different model sizes.
Anti-repetition: validation loss of MGA-trained models stays lower than the baseline under repeated training cycles, indicating better resistance to repetition degradation.