When Exponential Meets Power Law: Why 'Large Language Monkeys' Scaling Laws Hide Heavy-Tailed Distributions

> Feynman once said: "If you can't explain your research to a stranger at a bar, you haven't really understood it." The story here is about a pattern that puzzled researchers — and the answer hides in the tails of probability distributions.

The Puzzle

Researchers recently observed a strange statistical regularity. When a large language model gets multiple attempts per problem — math, code, jailbreak attacks, where one success counts — the overall success rate follows a power law:

In plain terms: gains from more attempts get smaller and smaller. Big improvement from the first try, but by attempt 100 or 1000, each extra attempt barely helps. This is a well-known "scaling law."

But there's a problem. Simple math predicts that for each *individual* problem, the failure rate should decay exponentially with attempts. Exponential decay means fast convergence: if the first-attempt success rate is 10%, by attempt 32 it's near 97%.

Exponential per problem, power law in aggregate. How can both be true?

A Feynman-Style Thought Experiment

Imagine 1000 monkeys with typewriters. Some target sentences are easy ("Hi"), some moderate ("To be or not to be"), some nearly impossible (a Shakespearean sonnet).

For any single monkey, the probability of eventually typing its target grows exponentially: if the per-attempt success probability is \(p\), then success within \(k\) attempts is \(1 - (1-p)^k\), and \((1-p)^k\) decays exponentially toward zero.

But the aggregate picture differs. The monkeys facing near-impossible tasks — \(p \approx 0\) — essentially never succeed. In the pooled statistics, these extreme stragglers dominate the trend. That is the core of the paradox: individually exponential, collectively a power law.

The Math

Suppose there are \(N\) problems, where problem \(i\) has single-attempt success probability \(p_i\). After \(k\) attempts:

For any fixed \(p_i\), this is exponential scaling. The average failure rate across problems is:

The key: the result depends on the distribution of \(p_i\). If all \(p_i\) are similar, the sum decays exponentially. But if the distribution is heavy-tailed — most problems have moderate \(p_i\), with a small cluster of near-zero values — a few slow-decaying terms dominate. When \(p\) follows \(P(p \leq x) \sim x^\alpha\):

An aggregate power law emerges.

Empirical Validation

The researchers verified this in three domains: math problem solving, Lean proof generation, and multimodal jailbreak attacks.

• Per-problem is exponential: isolated problems show clean exponential failure decay, with high goodness of fit.
• Aggregate is power law: pooled success vs. attempts forms a straight line on a log-log plot.
• The distribution is heavy-tailed: most problems have single-attempt success rates of 10%–50%, but a long tail extends below 0.0001%. These extremely hard problems control the aggregate at large attempt counts.

This perspective also explains earlier anomalies:

• Benchmarks that don't obey the power law lack a sufficiently heavy-tailed difficulty distribution.
• Power-law exponents differ across tasks because their difficulty-tail thickness differs.

Why It Matters

Predicting scaling cheaply. Previously, forecasting performance at \(k\) attempts required actually running \(k\) attempts — expensive. Now you can run one attempt per problem, examine the success-probability distribution's tail, and estimate the power-law exponent directly. Per the paper, this predicts the exponent with roughly an order of magnitude lower relative error, requiring 2–4 orders of magnitude less inference compute.

Benchmark design. Reliable benchmarks should include a heavy tail of very hard problems; model comparisons should focus on the hardest items, since that's where capabilities truly diverge. Most benchmarks fail to separate frontier models because they lack difficult-enough problems — every model exceeds 90%, leaving too short a tail.

The Takeaway

Feynman might have put it this way: two seemingly contradictory macroscopic patterns — exponential and power law — become perfectly consistent once you understand the heavy-tailed structure underneath. When you see an aggregate statistical law, look at the shape of the distribution. The truth is often not in the mean, but in the tail.

---

*Paper info*

• Title: How Do Large Language Monkeys Get Their Power (Laws)?
• Authors: Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, Sanmi Koyejo
• Venue: ICML 2025 (Oral)
• Link: OpenReview
• Code: GitHub