UnpredictaBench: LLMs Fail at Generating Random Distributions — No Model Exceeds 40% Accuracy

Forum topic · 小凯 · 2026-06-09

Summary

UnpredictaBench is a benchmark from University of British Columbia researchers evaluating how faithfully large language models generate random samples from probability distributions. Spanning 448 test instances, 40 classical distributions, and 7 task types (explicit/implicit text, explicit/implicit code, multimodal distributions, list shuffling, and real-world scenarios), it uses KS@N — a metric based on the Kolmogorov-Smirnov test — to measure distribution match. The findings are stark: under default settings (KS@100), no model surpasses 40% accuracy, and most closed-source models like GPT-4 and Claude 3.5 score only 0-20%. Some small open-source models outperform larger proprietary ones, suggesting randomness is not a monotonic function of scale and that alignment training may suppress genuine sampling diversity. Typical failure modes include mode collapse toward central values, drifting to wrong distribution shapes, incorrect code parameters, and near-zero accuracy on list shuffling. The post argues the root cause is architectural: autocratic token-by-token sampling, RLHF conservatism, and token-sequence representations of numbers make faithful probabilistic sampling fundamentally hard. It recommends using dedicated numerical tools (NumPy, SciPy, R) for simulations rather than LLMs.

Overview

A recent post on zhichai.net discusses UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs (Amirhossein Abaskohi et al., University of British Columbia + independent researchers), available at https://arxiv.org/abs/2606.06622. The central finding: when asked to generate random samples from probability distributions, no LLM exceeds 40% accuracy under default settings, and most models sit in the 0-20% range.

Why Randomness Matters

Faithful sampling from probability distributions underpins:

Monte Carlo simulation (finance, drug screening, physics)
Synthetic data generation and augmentation
Games, creative applications, procedural generation
Scientific computing: Bayesian inference, A/B test design

Prior evaluation relied on subjective human inspection or "LLM-as-judge" approaches — neither statistically rigorous nor reproducible.

Benchmark Design

448 test instances, 40 distributions, 7 task types, covering unimodal, multimodal, discrete, bounded, heavy-tailed, and special distributions
Tasks: explicit text prompts, implicit text (infer the distribution), explicit code, implicit code, multimodal sampling, list shuffling, real-world scenarios
KS@N metric: run a Kolmogorov-Smirnov test between N model-generated samples (default N=100) and true distribution samples; pass if p > 0.05. Requires only samples from the true distribution — no closed-form density needed — making evaluation automated, scalable, and reproducible.

Key Findings

No model passes 40%; GPT-4 and Claude 3.5 score 0-20%; some small open-source models outperform GPT-4o, implying randomness capability is not monotonic with scale
Alignment training may hurt randomness — RLHF pushes models toward "safe, average, non-extreme" outputs
Failure modes: mode collapse (samples squeezed into a narrow interval), distribution drift (e.g., continuous decimals for a Poisson distribution, no extreme values for Cauchy), broken code (wrong parameter order, wrong functions), and near-0% on uniform list shuffling
Chain-of-thought helps only marginally — the bias lies in the underlying sampling mechanism, not reasoning

Root Causes (as analyzed in the post)

1. Autoregressive decoding (greedy, temperature, top-k/top-p) is optimized for fluent text, not faithful distribution reproduction 2. RLHF trains away extreme values 3. Training corpora rarely contain true random samples 4. Numbers are token sequences, not floats — generation is discrete sequence prediction, not numerical sampling

Practical Takeaways

Don't use LLMs for precise sampling: use NumPy/SciPy/R for Monte Carlo, simulations, and A/B testing
Manually verify statistically generated code (parameters, function choice)
LLMs are suited to *explaining* randomness, not *executing* it
KS@N-style distribution-level evaluation could extend to image, music, and molecule generation; LLM-as-judge paradigms deserve scrutiny given judges score ~20% on this task themselves

Conclusion

The post concludes the limitation is mechanistic, not a matter of scale: autoregressive language models were never designed for faithful probabilistic sampling. Next-generation architectures may need external tools, neuro-symbolic hybrids, or probabilistic programming interfaces for true randomness.