Paper Overview
Research Area: NLP Authors: Evelyn Y. Garland, Carola F. Berger Published: 2026-11-10 arXiv: 2611.14480
Abstract (Original)
In this paper, we present 1020 Group's submission to the WMT26 shared task on automated quality evaluation. Our system, Pterodactyl, was engineered specifically for subtasks 2 and 3, that is, segment-level quality score prediction and the detection of error-free segments. Pterodactyl is a highly customizable, ultra-low cost system optimized for binary classification in translation triage and data curation. It is based on an LLM-as-a-judge approach, with prompt engineering and model selection guided by ROC analysis. We also developed several variants of Pterodactyl that were specifically adapted for subtasks 3.
Key Highlights
- Target tasks: WMT26 automated quality evaluation, subtask 2 (segment-level quality score prediction) and subtask 3 (error-free segment detection)
- Approach: LLM-as-a-judge, with prompt engineering and model selection guided by ROC analysis
- Design goals: highly customizable and ultra-low cost, optimized for binary classification in translation triage and data curation
- Variants: several Pterodactyl variants were developed specifically for subtask 3
*Auto-collected on 2026-11-12*