Does Jev-Style System One Structure Reduce Selection Bias in LLMs? A Systematic Study

Forum topic · 小凯 · 2026-10-27

Summary

Selection bias in large language models (LLMs) persists despite instructions to ignore presentation order and reasoning prompts. This paper systematically tests whether Jev-style System One models—a family that directly predicts probabilities over supplied candidates—reduce selection bias in multi-choice tasks. The authors shuffle option order while keeping question content fixed, tracing individual answers across eight presentations for two Jev versions, twelve Jev reproductions, and six generative LLMs. On 1,700 questions from two benchmarks, Jev accuracy varies by under two percentage points, yet its cumulative failure set grows 27–29% relative to the original order, with first failures persisting into the final presentation. This loss of initially correct answers far exceeds that under fixed-order repetition, even for highly repeatable Jev reproductions. Jev shows no general advantage over LLM baselines in preserving correct choices, and aggregation (voting) offers only limited protection, with even unanimous answers frequently wrong. The study exposes a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing that stable accuracy alone does not demonstrate resistance to selection bias.

Overview

Field: NLP
Authors: Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou
Published: 2026-10-26
arXiv: 2610.35826

Key points

Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias?

The authors address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, they trace individual answers through eight presentations and compare them with fixed-order repetition.

Findings

On the primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27–29% relative to the original order, with first failures continuing into the final presentation.
This loss of initial successes is substantially larger than under fixed-order repetition, even for highly repeatable Jev reproductions.
Jev shows no general advantage over the LLM baselines in preserving correct choices.
Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong.

Conclusion

These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.

--- *Auto-collected on 2026-10-28*