UnpredictaBench: A New Benchmark Shows LLMs Fail at Generating Random Distributions

Forum topic · 小凯 · 2026-06-09

Summary

A team from the University of British Columbia introduced UnpredictaBench, a benchmark for evaluating how well large language models generate samples from probability distributions. The benchmark contains 448 test instances covering 40 classical distributions across 7 task types, including explicit and implicit text generation, code generation, multimodal distributions, list shuffling, and realistic scenarios. Evaluation uses KS@N, a metric based on the Kolmogorov-Smirnov test that statistically compares model-generated samples against the true distribution. Results show that no model exceeds 40% accuracy at KS@100 under default settings, with most models scoring 0-20%. Notably, some small open-source models outperform larger closed-source models, suggesting randomness generation is not a monotonic function of scale and that alignment training may suppress true stochasticity. Typical failure modes include mode collapse toward central values, distribution drift, parameter errors in generated code, and near-zero accuracy on shuffling tasks. Chain-of-thought prompting yields only marginal improvements. The findings indicate that autoregressive token prediction and RLHF alignment are fundamentally mismatched with faithful probabilistic sampling, and practitioners should use dedicated numerical tools like NumPy or R for tasks requiring precise random sampling.

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs Paper: https://arxiv.org/abs/2606.06622 Authors: Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo, Ellie Dingqiao Wen, Lele Wang, Giuseppe Carenini, Peter West (University of British Columbia + independent researchers)

The Problem: AI Can't Even Roll Dice?

We're used to LLMs writing poetry, code, and reasoning through problems. But one seemingly trivial question trips them all up:

"Generate 10 random numbers from a normal distribution."

The UnpredictaBench benchmark from UBC exposes the truth:

No LLM under default settings exceeds 40% accuracy at generating random distributions. Most models struggle between 0-20%. Some powerful closed-source models perform worse than small open-source ones.

Why Randomness Matters

Monte Carlo simulation: financial risk, drug screening, physics simulation all rely on sampling from distributions
Data augmentation: synthetic data generation from specific distributions
Games and creativity: NPC behavior, level generation, music composition
Scientific computing: statistical inference, Bayesian methods, A/B test design

Previous evaluation methods were ad hoc: manual inspection (subjective, irreproducible) or LLM-as-judge (circular reasoning).

How UnpredictaBench Works

Test Library

448 test instances, 40 classical distributions across categories:

| Distribution Type | Examples | Difficulty | |---|---|---| | Unimodal | Normal, exponential, Poisson | Simple shape, tail drift | | Multimodal | Bimodal normal, Gaussian mixtures | Models hit only one mode | | Discrete | Bernoulli, binomial, geometric | Models output decimals | | Bounded | Uniform, Beta, triangular | Boundary violations | | Heavy-tailed | Cauchy, Pareto | Models avoid extreme values | | Special | Chi-square, F, t distributions | Parameter sensitivity |

Seven Task Types

1. Explicit text: "Generate 10 random numbers from N(0,1)" 2. Implicit text: infer the distribution from a scenario description 3. Explicit code: write code to sample a specified distribution 4. Implicit code: identify the distribution from a functional description 5. Multimodal distributions: samples with multiple peaks 6. List shuffling: uniformly random permutations 7. Realistic scenarios: e.g., simulating exam scores

KS@N Metric

1. Model generates N samples (default N=100) 2. Kolmogorov-Smirnov test against N samples from the true distribution 3. Pass if P-value > 0.05

Key advantages: statistically rigorous, reproducible, fully automated, requires only samples (no closed-form density), and evaluates at the distribution level rather than per-output.

Results: A Humbling Picture

| Model Type | KS@100 Accuracy | Notes | |---|---|---| | Best model (any) | <40% | No model passes | | Most closed-source models | 0-20% | | | Some small open-source models | 10-25% | Sometimes beat larger models | | Shuffling tasks | 0% (some models) | Even strong models fail |

Counterintuitively, small open-source models (e.g., Mistral-7B, Llama-3-8B) outperform GPT-4o on some tasks, suggesting randomness generation is not a monotonic function of scale, and alignment training may suppress genuine stochasticity.

Failure Modes

1. Mode collapse: all samples cluster in a narrow interval, ignoring tails 2. Distribution drift: wrong shape entirely — decimals for discrete distributions, missing heavy tails 3. Code reasoning doesn't save it: swapped parameters, wrong functions, syntactically correct but logically wrong code 4. Shuffling catastrophe: models produce "human-like pseudorandom" sequences rather than uniform permutations

Chain-of-thought prompting helps only marginally — the problem lies in the biased underlying sampling mechanism, not reasoning ability.

Why LLMs Fail at Randomness

1. Autoregressive sampling is biased: greedy decoding and Top-k/Top-p truncation are designed for fluent text, not faithful distribution reproduction 2. Alignment kills randomness: RLHF trains models to avoid extreme, "unreasonable" outputs — good for chat, disastrous for sampling 3. Training data lacks random samples: internet text rarely contains "here are 100 random numbers from N(0,1)" 4. Language isn't a container for numbers: "3.14159" is a token sequence, not a float; generation is discrete sequence prediction, not continuous sampling

Implications

For model developers: randomness is a capability blind spot; consider dedicated numerical sampling heads, distribution-focused training data, and scenario-aware alignment.

For practitioners: don't use LLMs for precise random sampling — use numpy/scipy/R. Verify generated statistical code manually. LLMs are good at *understanding* randomness, not *executing* it.

For evaluation: KS@N fills the gap of distribution-level evaluation and could extend to image, music, and molecular generation. When a judge model itself scores ~20% on randomness, LLM-as-judge credibility deserves scrutiny.

Conclusion

UnpredictaBench's value is threefold: a quantifiable, reproducible evaluation framework (KS@N); the first systematic benchmark for distributional randomness; and a direction for next-generation architectures — external tools, neuro-symbolic hybrids, and probabilistic programming interfaces. LLMs were never designed to faithfully sample probability distributions; when you need precise randomness, use Python, not GPT-4.