English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Vocabularies (arXiv 2611.08439)

Forum topic · 小凯 · 2026-11-07

Summary

Typed-decision models score options supplied by the caller at request time, so accuracy depends on label wording the model developer does not control. This paper compares two releases of Ru-Jev, a 307M-parameter multilingual cross-encoder trained on thirteen Russian-language tasks, where the later release adds data augmentation presenting labels in multiple surface forms. Holding questions fixed and varying only the wording of eighteen routing labels, the authors evaluate ten label vocabularies generated by three separate processes, with five candidates per question (eighteen for eight of them). On held-out vocabularies of ordinary or borrowed words (English-derived jargon, office slang, department names), the augmented release is more accurate in every setting, by up to 0.24; eleven of fourteen gains survive a Holm correction. Abbreviations and opaque codes show no usable gain, with both releases at or below chance on opaque codes once training-related labels are removed. Risk-coverage area falls in most improved cells, though confident errors rise with eighteen candidates on jargon and slang. Accuracy shifts by -8.4 and +5.6 points on two unseen datasets, and the authors report four evaluation hazards when comparing releases.

论文概要

研究领域: NLP 作者: Vladimir Beskorovainyi 发布时间: 2026-11-05 arXiv: 2611.08439

Abstract

Typed-decision models score options that the caller supplies at request time, so their accuracy depends on label wording that the model's developer does not control. We compare two releases of a 307M-parameter multilingual cross-encoder, Ru-Jev, trained on thirteen Russian-language tasks; the later release adds an augmentation that presents labels in several surface forms. Holding questions fixed and varying only how eighteen routing labels are worded, we evaluate ten label vocabularies, produced by three separate processes and checked for overlap with the training data, with five candidates per question and, for eight of them, also with eighteen.

On held-out vocabularies made of ordinary or borrowed words (English-derived jargon, office slang, department names), the augmented release is more accurate in every setting, by up to 0.24. Eleven of these fourteen gains survive a Holm correction (slang with eighteen candidates and descriptive sentences with five do not), and for jargon on all labels the label-level interval excludes zero in both settings.

On abbreviations and opaque codes it gives no usable gain: on abbreviations it is not better, and worse on the clean slice, and on opaque codes both releases stay at or below chance once labels related to training options are removed. In most cells where accuracy rises, the risk-coverage area falls, but with eighteen candidates confident errors rise on jargon and slang. On two datasets neither release has seen, accuracy moves by -8.4 and +5.6 points. We also report four evaluation hazards from comparing the two releases.

Key Findings

  • Label-wording augmentation improves robustness on held-out vocabularies of ordinary and borrowed words, with gains up to 0.24.
  • 11 of 14 gains survive Holm correction; jargon shows statistically significant improvement in both candidate settings.
  • No usable gains on abbreviations or opaque codes; both model versions perform at or below chance on opaque codes without training-related labels.
  • Risk-coverage area decreases in most improved settings, but confident errors increase with 18 candidates on jargon and slang.
  • Generalization to unseen datasets is mixed: -8.4 and +5.6 accuracy points.
  • The paper documents four evaluation hazards when comparing two model releases.
--- *Source: arXiv:2611.08439*

Tags

#nlp#cross-encoder#data-augmentation#label-wording#arxiv#multilingual-models#russian-language

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636943