Last Translation Benchmark: Human-Authored, Peer-Reviewed Examples That Break Leading Machine Translation Models

Forum topic · 小凯 · 2026-09-07

Summary

The Last Translation Benchmark (LTB) is a new evaluation resource for machine translation, introduced by Vilém Zouhar, Niyati Bafna, and Mukund Choudhary (arXiv:2509.04283). The paper argues that standard MT benchmarks are approaching saturation as models improve, while automatic translation metrics remain unreliable, vulnerable to reward-hacking, and unactionable; even gold human evaluation suffers from limited reproducibility, objectivity, and scalability. LTB addresses these gaps with a collection of human-authored, peer-reviewed examples spanning text, images, audio, and video, designed specifically to break leading machine translation models. Crucially, each example is paired with handcrafted verification rules describing concrete failure cases, enabling reliable and actionable evaluation. LTB is a living dataset accepting continuous contributions; its first release, LTBv1, includes contributions accepted before September 1, 2026, with periodic updates planned as new data arrives.

Paper Overview

Field: NLP / Machine Translation
Authors: Vilém Zouhar, Niyati Bafna, Mukund Choudhary
Published: 2026-09-06
arXiv: 2509.04283

Abstract (English)

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, vulnerable to reward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation methodology: each example comes with handcrafted verification rules that describe specific failure cases on that example, enabling reliable and actionable assessment. The Last Translation Benchmark is a living dataset that accepts continuous contributions. The latest version is LTBv1, containing contributions accepted before September 1, 2026, with updates planned as new data is continuously collected.

Key Points

Benchmark saturation: Standard machine translation benchmarks no longer effectively discriminate between state-of-the-art models as capabilities improve.
Metric limitations: Automatic translation metrics are unreliable and susceptible to reward-hacking, and their outputs are hard to act on.
Human evaluation limits: Gold human evaluation lacks reproducibility, objectivity, and scalability.
Proposed solution: The Last Translation Benchmark — human-authored, peer-reviewed examples across text, image, audio, and video modalities that break leading MT models.
Novel evaluation: Each example includes handcrafted verification rules describing concrete failure modes, making assessments reliable and actionable.
Living dataset: LTB accepts ongoing contributions; LTBv1 covers submissions accepted before 2026-09-01, with regular updates planned.