Last Translation Benchmark: Human-Curated Examples That Break Machine Translation Models

Forum topic · 小凯 · 2026-09-07

Summary

The Last Translation Benchmark (LTB) is a new evaluation resource for machine translation, introduced by Vilém Zouhar, Niyati Bafna, and Mukund Choudhary (arXiv:2509.04283). It addresses two problems in MT evaluation: standard benchmarks are approaching saturation as models improve, and automatic metrics are unreliable, vulnerable to reward hacking, and provide unactionable assessments, while gold human evaluation lacks reproducibility, objectivity, and scalability. LTB is a collection of human-authored, peer-reviewed examples spanning texts, images, audio, and video, specifically designed to break leading machine translation models. The authors also propose a new evaluation methodology: each example comes with handcrafted validation rules describing specific failure cases, enabling reliable and actionable evaluation. LTB is a living dataset that accepts continuous contributions. The current release, LTBv1, includes contributions accepted before September 1, 2026, with planned updates as new data is collected.

Overview

Field: NLP (Machine Translation) Authors: Vilém Zouhar, Niyati Bafna, Mukund Choudhary arXiv: 2509.04283

Abstract

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, vulnerable to reward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement.

We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation methodology: each example is accompanied by handcrafted validation rules that describe specific failure cases for that example, enabling reliable and actionable evaluation.

Key Points

Benchmark saturation: Standard MT benchmarks no longer differentiate between top models as capabilities improve.
Metric limitations: Automatic translation metrics are unreliable and susceptible to reward hacking, offering unactionable feedback.
Human evaluation issues: Gold human evaluation often lacks reproducibility, objectivity, and scalability.
Multimodal coverage: The benchmark includes texts, images, audio, and video examples.
Quality control: All examples are human-authored and peer-reviewed.
Actionable evaluation: Handcrafted validation rules per example describe concrete failure modes.
Living dataset: LTB accepts continuous contributions; LTBv1 contains contributions accepted before September 1, 2026, with updates planned as new data is collected.

---

*Auto-collected on 2026-09-07.*