Overview
Paper: *OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment* Authors: Cho Seonglae, Koshiyama Adriano arXiv: https://arxiv.org/abs/2607.26981
A core question: if an LLM says a startup has a 70% chance of success and a 15% chance of failure, where do the missing 15 percentage points go? The gap signals a directional bias that calibration metrics miss, because ECE and Brier score cancel positive and negative errors. OptimismBench detects this bias without any ground-truth labels.
Method: Inverted Pairs
The benchmark does not compare predictions to real outcomes. It compares a model's answers to two phrasings of the same scenario:
- Version A: "What is P(success)?"
- Version B: "What is P(failure)?"
- Skew > 0 → optimistic bias
- Skew < 0 → pessimistic bias
- Track A (calibration control): 15 items with known base rates
- Track B (probability estimation): the inverted-pair main track
- Track C (recommendation): replacing probability questions with recommendations
- Track D (salience): varying how the scenario is described
- Structural blind spot in calibration metrics. Unsigned metrics absorb directional drift, so a well-calibrated model can still systematically tilt predictions upward.
- Alignment is not neutral. RLHF and DPO shape not only safety and helpfulness but also the sign and magnitude of forecasting bias, with direct implications for risk assessment, medical triage, and financial prediction.
- Downstream pipelines inherit the tilt. The paper's closing line: when alignment makes a model more helpful, it also tilts its probabilities, and downstream pipelines inherit the tilt by default.
- Family-level direction claims rest mainly on Qwen and Llama pairs; Gemma-2-2b and Mistral-Small-24B are single-pair pilots.
- Cross-language evidence is uneven across 6 native prompts and 4 English system prompts (DE, FR, HI, JA).
- Skew measures internal consistency, not accuracy against human judgment.
- No matched human-subject experiment; human data is cited from the literature.
- Paper: https://arxiv.org/abs/2607.26981
- HTML: https://arxiv.org/html/2607.26981v1
- Dataset: 3,870 items across 10 languages, open-source
Under complementarity, P(success) + P(failure) = 100%. Any shortfall is a Skew measure:
Four auxiliary tracks probe boundary conditions:
Four interventions—narrative manipulation, perspective shift, anchoring gradient, and self-debiasing—were tested. The bias survived every intervention.
Key Findings
1. 14 of 16 models are optimistic. The two pessimistic models are Claude variants from Anthropic's frontier line. 2. Post-training sets the sign of bias. Eleven base-vs-chat pairs across four families show that the same base model can flip its bias direction depending on the provider's post-training. Qwen flips from pessimistic to optimistic; Llama flips from optimistic to pessimistic. 3. Model identity dominates language. Across 17 models in 6 languages, between-model variance was 4.7× between-language variance. Language is a secondary factor. 4. Bias reverses under recommendation framing. An optimistic probability judge can become pessimistic when asked for recommendations, meaning directional bias is a joint property of (model, task framing), not a fixed model trait.
Why This Matters
Cross-Domain Analogy
Human optimism bias (Weinstein, 1980) is a structural parallel: models appear to have learned human judgment patterns, including human cognitive biases. A key difference is that human bias is broadly optimistic, while LLM bias direction is selected during post-training, raising the question of whether alignment should target neutrality or human-like optimism. The benchmark now provides a way to measure the choice.