[论文] [论文] Understanding Jev-Style System One Models: Does Structure Reduce ...

论文概要

研究领域: NLP
作者: Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou
发布时间: 2026-10-26
arXiv: 2610.35826

中文摘要

LLM 中的选择偏差可无视「忽略呈现顺序」的指令，也不因引入推理提示而消失。近期复现 Jev 式「系统一」行为的工作产生了一个直接对候选计算预测概率的模型族，但多选项任务中跨 Jev 版本、Jev 复现与生成式 LLM 的选择偏差系统比较仍然缺乏，核心问题悬而未决：Jev 式决策结构能否降低选择偏差？我们用简单测试回答：固定问题与候选内容，打乱选项顺序。在两个 Jev 版本、十二个 Jev 复现、六个生成式 LLM 上，追踪单个答案在八种呈现下的变化，并与固定顺序重复比较。在主要来自两个基准的 1700 题主集合上，Jev 准确率波动不到 2 个百分点，但累计失败集合比原始顺序扩大 27%–29%，首次失败持续到最后一轮呈现；这种初始正确答案的流失远大于固定顺序重复，即便高度可复现的 Jev 复现也如此。Jev 在保持正确选择上对 LLM 基线也无普遍优势。聚合保护有限：投票对 Jev 收益甚微，一致同意答案也经常错。结果揭示了「可重复的决策」与「等价呈现下仍正确的决策」之间的鸿沟——稳定准确率不能证明对选择偏差的抵抗力。

原文摘要

Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias? We address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, we trace individual answers through eight presentations and compare them with fixed-order repetition. On our primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27--29% relative to the original order, with first failures continuing into the final presentation. This loss of initial successes is substantially larger than under fixed-order repetition, even for highly repeatable Jev reproductions. Nor does Jev show a general advantage over the LLM baselines in preserving correct choices. Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong. These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.

自动采集于 2026-10-28

#论文 #arXiv #NLP #小凯