English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Type-Safe Is Not Error-Free: A Constrained Decision Head Still Fails When Option Names Change

Forum topic · 小凯 · 2026-09-24

Summary

A new arXiv paper (2609.26758) by Yu Sun and Junhao Xu examines typed decision models that output schema-conforming decisions over predefined options rather than free-form text. Although every output is type-safe by construction, the models may not interpret option names as intended. Testing Jev and two Jev-like open-weight models, the authors changed only the option names assigned to fixed rubrics. On 1,200 workflow decisions, renaming two options from 0/1 to no/yes flipped 70.4 answers per hundred (95% CI [67.6, 73.1]) and dropped AUC from 0.94 to 0.23—a systematic reversal of decision ranking. Neutral names had almost no effect; the pattern held across all 4 predicates (at least 7.4x the neutral control), strengthened with more options, and depended on read-out geometry (mean-pooling models flipped 4.1x less). A hosted model showed the same behavior (AUC 0.8146 to 0.5806; 24x more flips than its test-retest floor). Replacing option names with random strings restored neutral behavior without accuracy loss, showing the failure stems from semantic polarity of option names, not renaming itself. Type-error rates stayed 0% throughout.

Paper Overview

Field: LLM Evaluation Authors: Yu Sun, Junhao Xu Published: 2026-09-22 arXiv: 2609.26758

Abstract (Original)

Typed decision models are built for settings where model outputs are consumed directly by software. Instead of generating free-form text, they return a decision over a predefined set of options. By construction, every output conforms to the required schema. Yet this guarantee does not tell us whether the model interprets the options as intended.

We study Jev and two Jev-like models with open weights by changing how option names are assigned to rubrics. Each option consists of an option name and a textual rubric that defines what the option means. We change only which option name is assigned to each rubric; the question, state, rubric wording, and set of option names remain exactly the same.

On 1200 workflow decisions with task-specific rubrics, renaming the two options from 0/1 to no/yes changes 70.4 more answers per hundred (95% CI: [67.6, 73.1]) and shifts AUC from .94 to .23, revealing a systematic reversal in the decision ranking rather than simple uncertainty. The same operation has little effect with neutral option names.

This pattern holds across all 4 predicates, where the effect is at least 7.4x larger than under the neutral control, and becomes stronger as the number of options increases. The effect also depends on the read-out geometry: a second model family that mean-pools over the full option span flips 4.1x less often. The hosted model exhibits the same behavior: the swap changes AUC from .8146 to .5806 and produces 24x as many answer flips as its test-retest floor.

In contrast, replacing the option names with random character strings returns all model families to the neutral-control regime without reducing accuracy. The failure therefore depends on the semantic polarity of the option names rather than on the renaming operation itself. Across all conditions, the type-error rate remains 0%, even when decision accuracy degrades substantially.

Key Findings

  • Type safety ≠ correctness: schema-conforming outputs can still encode systematically wrong decisions.
  • Semantic polarity matters: renaming 0/1 to no/yes flipped 70.4 answers per hundred and inverted AUC (0.94 → 0.23).
  • Consistent across predicates: effect is ≥7.4x larger than neutral control on all 4 predicates; grows with option count.
  • Read-out geometry matters: mean-pooling over the full option span reduces flips 4.1x.
  • Hosted models affected too: AUC 0.8146 → 0.5806; 24x more flips than test-retest floor.
  • Random names are harmless: random strings restore neutral behavior without hurting accuracy—failure depends on semantic polarity, not renaming itself.
*Auto-collected on 2026-09-24.*

Tags

#llm-evaluation#typed-decision-models#robustness#schema-constrained-output#arxiv-paper#machine-learning#model-evaluation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178635149