Your LLM Leaderboard Might Be Fake: A Self-Audit of Evaluation Reproducibility

A Question That Should Unsettle Every Evaluator

Say you ran an LLM evaluation: 8 models, 22 prompt files, 293 raw intermediate representations. You produced a leaderboard showing model A on top and model H at the bottom, and wrote a paper.

Now someone asks: if you reran the experiment, would the leaderboard be the same?

This is not a theoretical question. Dipankar Sarkar, in the paper *How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure*, audited his own earlier study and found troubling results:

• Only the bottom of the leaderboard is stable: the two worst models kept their ranks in 99% and 86% of resamples.
• The four middle models: rank stability of only 27%-48%.
• The top two models: rank stability of only 68%.
• Two equally defensible aggregation rules changed 4 of 8 rows and shifted a study-level conclusion by 7 percentage points.
• 4 of 8 endpoints were taken offline within 10 weeks of measurement-reproduction was simply impossible.

The leaderboard you see may be just one random sample from many possible leaderboards.

What the Experiment Did

The original study was an LLM structure-inference task: given a prompt file, the model had to infer its structural rules (e.g., "what this paragraph is doing," "here is a conditional branch"). 8 open-weight model variants from 5 families, from 8B to 675B parameters.

The core method of the self-audit is the joint cluster bootstrap: treat the 22 prompt files as samples, resample them with replacement, and recompute the leaderboard each time. Repeat many times and observe rank stability.

It is like the margin of error in a poll: ask 1,000 people and you get a support rate. Resample another 1,000 and the rate changes. The bootstrap answers exactly that question-just with prompts instead of people, and ranks instead of approval ratings.

The "Iceberg Effect" of Leaderboards

The results reveal what I would call an "iceberg effect":

The bottom of the leaderboard is stable; the top and middle are not. The two worst models land in the last two spots almost every time, while the middle four and top two fluctuate wildly across resamples.

It's like class exam rankings: the two students who always skip class reliably occupy the bottom. But the order of ranks one through six may change every exam.

This has deep implications for LLM evaluation:

1. You can reliably identify "bad models": if a model sits at the bottom, you can be fairly confident it really is bad. 2. You cannot reliably identify "good models": the #1 rank may just be lucky on your particular prompt sample. 3. Middle ranks carry almost no information: the gap between #3 and #6 may be pure noise.

In practice, however, what we care most about is "which model is best"-precisely the least stable part.

Aggregation Rules: An Overlooked Decision Point

One of the paper's most unsettling findings: two equally reasonable aggregation rules changed 4 of 8 leaderboard rows.

In the original study, some prompts were run multiple times ("campaigns"). When merging repeated results, you have two choices:

• Rule A: average within each campaign first, then average across campaigns.
• Rule B: pool all raw campaign data and average directly.

Both rules are mathematically "reasonable," yet they produce different leaderboards-4 of 8 rows change ranks. The study-level headline conclusion ("model X outperforms model Y by Z percentage points") shifts by 7 points.

This means: your conclusions partly depend on an analytical choice you may not even realize you made. Rule A yields one number; Rule B yields another. Both are "correct," but they point to different conclusions.

Non-Reproducibility After 10 Weeks

An even more discouraging finding: 4 of the 8 model endpoints were decommissioned or modified within 10 weeks of measurement.

This is not the author's fault-it is inherent to LLM evaluation. Providers (OpenAI, Anthropic, Google, etc.) update model versions irregularly. The "gpt-4o-2024-08-06" you measure today may be gone next month-replaced by a new version, or the API endpoint simply closed.

So: LLM evaluation papers have an expiration date. A leaderboard published today may be unreproducible in 10 weeks-not because the method is flawed, but because the models themselves no longer exist.

This contrasts sharply with traditional science. Physicists can reproduce the century-old Michelson-Morley experiment and get the same result. But the "experimental subjects" of LLM evaluation change constantly-you never step into the same river twice.

Reproducible Does Not Mean Valid

The paper makes clear: reproducibility is a necessary but not sufficient condition for validity.

A leaderboard can be perfectly reproducible (same ranks every rerun) yet still measure the wrong thing. If all prompts share the same bias, the leaderboard will be very stable-stably measuring the wrong capability.

The paper recommends reporting the following so readers can judge for themselves:

1. Rank stability: bootstrap confidence intervals for rank positions. 2. Per-cell provenance: which prompts, campaigns, and time points produced each data point. 3. Sensitivity comparisons: how results change under different aggregation rules and analytical choices. 4. Raw per-run outputs: not just averages, but every individual run. 5. Measurement dates: an "expiration label" for the leaderboard.

The core message: treat a leaderboard as an estimate with error bars, not a definitive answer.

Statistical Instability vs. Study Defects: Two Different Failures

The paper draws an important distinction:

• Statistical instability: rank fluctuation caused by too small a prompt sample. Can be mitigated by adding more prompts.
• Study defects: systematic bias from methodological choices (aggregation rules, tie handling). Adding prompts cannot fix this.

These failures require different remedies. Statistical instability improves with "run more prompts"; study defects require re-examining the analysis pipeline. The original study suffered from both: a small prompt sample and a questionable aggregation rule. Fixing only one is not enough.

Practical Recommendations for the LLM Evaluation Community

1. Report bootstrap confidence intervals. Don't publish point-estimate leaderboards; report rank stability at each position. 2. Test aggregation-rule sensitivity. If you merge multiple runs, report results under at least two rules. 3. Report endpoint availability. Note each model's access method, version number, and measurement date. Readers need to know when the leaderboard "expires." 4. Distinguish "reproducible" from "valid." Reproducible means rerunning gives the same result; valid means the result measures what you intended. Both matter, and they must not be conflated. 5. Treat leaderboards as interval estimates, not point estimates. Model A at #1 does not mean it is truly better than #2-it may just have been lucky on your prompt sample.

Conclusion: A Leaderboard Is a Measurement, Not a Judge

The paper's deepest contribution is not a specific technical finding but an attitude shift: a leaderboard is not a judge, it is a measuring instrument. Measuring instruments have error bars, operating conditions, and shelf lives.

Next time you see an LLM leaderboard-MMLU, HumanEval, or any custom benchmark-remember:

• The bottom ranks may be reliable (bad models are genuinely bad).
• The top ranks may be unstable (the "best" model may just be lucky).
• Middle ranks carry almost no information.
• Aggregation rules may have changed half the leaderboard.
• In 10 weeks, half the models may no longer exist.

This does not mean leaderboards are useless. They are useful measuring instruments-but you need to know their error bars, just as you need to know a thermometer's precision. A thermometer accurate to ±5 degrees telling you "25 degrees today" means the real temperature is between 20 and 30. A leaderboard without stability reporting telling you "model A is best" might come with a 40% chance that the actual ranking is otherwise.

---

Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure