KletterMix: A 725B-Token German Pretraining Corpus Built by Translating ClimbMix

KletterMix takes a bold approach to German pretraining data: instead of crawling the German web, it translates the curated English corpus ClimbMix entirely into German, preserving its document structure, cluster mixture design, and topical diversity — 725B tokens produced in ~10 days on 126 nodes × 8 B200 GPUs.

Key points

• Why native crawling falls short: German poses structural challenges — heavy compounding (e.g., *Notarztwagenbesatzung*), rich inflection (3 genders, 4 cases), noun capitalization, and regional variants. Existing German resources (GermanWeb, LLäMmlein, German Commons, LeoLM, Occiglot, Teuken) suffer from heuristic filtering, limited scale, or insufficient native German content. The paper argues native crawling "cannot automatically reproduce the source diversity, mixture design decisions, and document standards of the strongest English pretraining datasets."
• Translation transfer strategy: Rather than translating arbitrary English text, KletterMix translates an already-curated pretraining corpus, inheriting a validated data distribution instead of re-deriving mixture ratios from scratch.

Engineering highlights

Length-aware routing (8 buckets)

Documents are routed into buckets (4k to over-64k) with different batch sizes, concurrency, and timeouts: short documents get high throughput (batch 3,072, 1,536 concurrent); long documents trade concurrency for stability. The dynamic target budget is:

\[\ell_{\max}^{\text{tgt}} = \max\left(2048,\ \min\left(32768,\ \lceil 2.0 \cdot \ell_{\text{src}} + 1024 \rceil\right)\right)\]

The 2.0 factor accounts for German text expansion; 32,768 is a hard cap.

Document-preserving translation

Sentence-level greedy packing into 20k-token source chunks, with a 2,000-token German context window carried over between chunks for discourse continuity. Prompt constraint: "Only output the German translation" — guarding against explanations or source repetition. Full metadata (document_id, source_cluster, context_bucket, quality_estimation, etc.) is retained for auditability.

Two-tier quality filtering

• Teacher: COMETKiwi quality labels on stratified samples only.
• Student: a gradient-boosting regressor using 13 target-side German features (GlotLID language ID, text morphology, lexical diversity); weighted Pearson = 0.735, Spearman = 0.733, MAE = 0.0477.
• Ablations tested thresholds 0.50/0.55/0.60 vs. unfiltered.

Results (Qwen3-0.6B, 12B tokens, 5-shot)

| Run | MMLU | PIQA | HellaSwag | ARC-C | Core Avg | |:---|:---:|:---:|:---:|:---:|:---:| | GermanWeb | 30.0 | 63.0 | 31.2 | 23.1 | 36.8 | | FineWeb2-DE | 28.7 | 70.0 | 31.5 | 23.0 | 38.3 | | KletterMix-Filt 0.60 | 28.5 | 70.0 | 34.6 | 27.5 | 40.2 |

• KletterMix shows lower pretraining and validation loss throughout — better sample efficiency and generalization, not overfitting.
• Gains are non-uniform: strongest on HellaSwag (+3.1) and ARC-C (+4.4) (coherence, structured reasoning), while native data retains advantages on MMLU (factual knowledge) and PIQA.

Implications for non-English pretraining

1. Translation transfer vs. native crawling is not either/or — translated data provides structure and explanatory content; native data provides facts, commonsense, and cultural specificity. A hybrid is likely optimal. 2. Filtering matters more than volume: strict filtering lifted Core Avg from 38.7 to 40.2, but thresholds are task-dependent. 3. The translation model is the ceiling: a Qwen3.5-397B-class model was required — a resource barrier for low-resource languages. 4. Document structure preservation matters, plausibly explaining the coherence/reasoning gains.

Limitations

• Cultural bias: English-world knowledge distribution; German-specific content may be underrepresented.
• Translationese effects not explicitly tested.
• The target-side-only filter cannot detect semantic drift.
• Anomalous "short translations" in long-source buckets need auditing.
• Cost: ~10,080 GPU-days.

Conclusion

KletterMix's core contribution is methodological: high-quality English pretraining corpus design can be transferred to non-English languages via a carefully engineered translation and filtering pipeline — but it requires a strong translation model, large compute, and native-data supplementation for factual and cultural coverage.

---

Reference: Kraus et al., "KletterMix: Climbing Toward High-Quality German Pretraining Data", arXiv:2606.03773, 2026. Source corpus: ClimbMix. Translation model: Qwen3.5-397B-A17B-FP8. Baselines: FineWeb2-DE, GermanWeb.