Comparative Reversal Learning Reveals Rigid Adaptation in LLMs Under Non-Stationary Uncertainty

Forum topic · 小凯 · 2026-04-07

Summary

This arXiv paper (2503.1384) by Haomiaomiao Wang, Tomás E Ward, and Lili Zhang evaluates large language models as sequential decision policies in a two-option probabilistic reversal-learning task with three latent states, where switch events are triggered by a performance criterion or timeout. Comparing a deterministic fixed transition cycle with a stochastic, more volatile schedule, the authors test DeepSeek-V3.2, Gemini-3, and GPT-5.2 against human behavioral data. Across all models, win-stay behavior was near ceiling while lose-shift was markedly attenuated, revealing asymmetric use of positive versus negative evidence. DeepSeek-V3.2 showed extreme perseveration after reversals and weak acquisition, while Gemini-3 and GPT-5.2 adapted faster but remained less loss-sensitive than humans. Stochastic transitions amplified reversal-specific perseveration in LLMs without uniformly reducing total wins, indicating high overall reward can coexist with rigid adaptation. Hierarchical reinforcement learning fitting suggests separable mechanisms behind rigidity: weak loss learning, inflated policy certainty, or value polarization through counterfactual suppression. The work motivates reversal-sensitive diagnostics and volatility-aware evaluation of LLMs under non-stationary uncertainty.

Paper Overview

Field: Machine Learning Authors: Haomiaomiao Wang, Tomás E Ward, Lili Zhang Published: 2025-04 arXiv: 2503.1384

Abstract

Non-stationary environments require agents to revise previously learned action values when contingencies change. We treat large language models (LLMs) as sequential decision policies in a two-option probabilistic reversal-learning task with three latent states and switch events triggered by either a performance criterion or timeout. We compare a deterministic fixed transition cycle to a stochastic random schedule that increases volatility, and evaluate DeepSeek-V3.2, Gemini-3, and GPT-5.2, with human data as a behavioural reference.

Key Findings

Asymmetric evidence use: Across models, win-stay was near ceiling while lose-shift was markedly attenuated, revealing asymmetric use of positive versus negative evidence.
Model differences: DeepSeek-V3.2 showed extreme perseveration after reversals and weak acquisition, whereas Gemini-3 and GPT-5.2 adapted faster but were still less loss-sensitive than humans.
Volatility effects: Stochastic transitions amplified reversal-specific perseveration in LLMs but did not uniformly reduce total wins — high overall reward can coexist with rigid adaptation.
Mechanistic accounts: Hierarchical reinforcement learning (RL) fitting suggests separable mechanisms behind rigidity: weak loss learning, inflated policy certainty, or value polarization through counterfactual suppression.

Implications

These results motivate reversal-sensitive diagnostics and volatility-aware models for evaluating LLMs under non-stationary uncertainty.

--- *Auto-collected on 2026-04-07*