English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Pterodactyl: 1020 Group's Efficient LLM-as-a-Judge System for WMT26 Quality Estimation

Forum topic · 小凯 · 2026-11-12

Summary

Pterodactyl is the submission by 1020 Group to the WMT26 shared task on automated quality evaluation, described in arXiv paper 2611.14480 by Evelyn Y. Garland and Carola F. Berger. The system targets subtasks 2 and 3: segment-level quality score prediction and detection of error-free translation segments. Pterodactyl is designed as a highly customizable, ultra-low-cost solution optimized for binary classification tasks in translation triage and data curation. It employs an LLM-as-a-judge approach, where prompt engineering and model selection are guided by ROC analysis to maximize classification performance. The authors also developed several Pterodactyl variants specifically adapted for the error-free segment detection subtask. This forum post summarizes the paper with both a Chinese abstract and the original English abstract, noting the research area as NLP and providing the arXiv link.

Paper Overview

Research Area: NLP Authors: Evelyn Y. Garland, Carola F. Berger Published: 2026-11-10 arXiv: 2611.14480

Abstract (Original)

In this paper, we present 1020 Group's submission to the WMT26 shared task on automated quality evaluation. Our system, Pterodactyl, was engineered specifically for subtasks 2 and 3, that is, segment-level quality score prediction and the detection of error-free segments. Pterodactyl is a highly customizable, ultra-low cost system optimized for binary classification in translation triage and data curation. It is based on an LLM-as-a-judge approach, with prompt engineering and model selection guided by ROC analysis. We also developed several variants of Pterodactyl that were specifically adapted for subtasks 3.

Key Highlights

  • Target tasks: WMT26 automated quality evaluation, subtask 2 (segment-level quality score prediction) and subtask 3 (error-free segment detection)
  • Approach: LLM-as-a-judge, with prompt engineering and model selection guided by ROC analysis
  • Design goals: highly customizable and ultra-low cost, optimized for binary classification in translation triage and data curation
  • Variants: several Pterodactyl variants were developed specifically for subtask 3
---

*Auto-collected on 2026-11-12*

Tags

#machine-translation#quality-estimation#llm-as-a-judge#wmt26#nlp#paper#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637076