Can LLMs Catch a Rigged Backtest? A Benchmark That Exposes Four Models' False Alarms

A Financial Engineer's Nightmare

Imagine you are a risk auditor at a quant fund. A researcher submits a strategy with a backtest showing 25% annualized returns, a Sharpe ratio of 2.1, and 8% max drawdown. Looks great.

But you suspect the backtest is rigged. Maybe the researcher tested on the training set (look-ahead bias), only picked stocks that never delisted (survivorship bias), or tuned parameters until the results fit (overfitting).

You decide to use an LLM to audit. You feed the strategy code to GPT-4o and ask, "What's wrong with this backtest?"

The LLM replies: "Detected look-ahead bias — training and test sets overlap."

You're satisfied. But have you considered: if an LLM says there's a problem, is there really one?

That is the core finding of a paper by Makar Ulesov et al. (MBZUAI + University of Pittsburgh): most LLM auditors are paranoid — they find real problems, but they also flag clean strategies as flawed.

Why the Experiment Design Is Clever

Most LLM-finance benchmarks measure "can the LLM find defects in a backtest?" This paper measures "can the LLM distinguish genuinely flawed from genuinely clean?"

The difference is calibration, not just recall.

The authors built 96 test items forming 48 pairs. In each pair:

• A flawed version: with a planted methodology error from 8 categories (look-ahead, survivorship bias, data snooping, etc.)
• A clean control: identical in strategy, dates, code style, labels, and report format — only one methodological detail differs

This means: if the LLM just "sounds the alarm whenever it sees look-ahead-related code," it will also flag clean versions — false positives. In the real world, every false alert means a phantom audit lead, wasted team time, and potentially masking the real problem.

The Numbers Are Chilling

The paper tested four models: DeepSeek V4 Flash, gpt-4o-mini, gemini-2.5-flash-lite, and gpt-4.1-mini. Under the "open code" condition (no hints, just "what's wrong with this code"):

| Model | Defect recall | False positives on clean strategies | Extra audit leads per 100 clean submissions | |---|---|---|---| | DeepSeek | 72.9% | 93.8% | 93.8 | | gpt-4o-mini | — | 38/48 | 79.2 | | Other models | similar | similar | similar |

Under open code, DeepSeek flagged 45 of 48 clean strategies. If you use it to audit 100 clean strategies, it will generate ~94 phantom audit leads.

You might think: just give the LLM hints. The paper tested three prompt conditions:

• Open: no hints
• Closed: definitions of the 8 defect categories
• Clean-aware: closed plus the note that "clean strategies are common"

Results on DeepSeek:

| Condition | Defect recall | False positive rate | |---|---|---| | open code | 72.9% | 93.8% | | closed code | 100.0% | 20.8% | | clean-aware code | 100.0% | 0.0% |

Clean-aware prompting is the only condition achieving 100% recall and 0% false positives. But this held only on DeepSeek. The other models kept high false positive rates even with clean-aware prompts.

"Recall Is Easy, Calibration Is Hard"

The paper's sharpest line: "Perfect recall is not complete audit quality."

In other words: finding all problems is not the same as high audit quality. If your auditor flags everything as problematic, recall is 100% — and it's useless, because you still have to manually check every flag.

This is the same as sensitivity vs. specificity in medical diagnostics. A test that diagnoses everyone as "sick" has 100% sensitivity and 0% specificity — clinically worthless.

The paper also measures a finer dimension: all-three specificity — simultaneously correct classification, correctly located evidence, and a correct fix suggestion. Under clean-aware code, DeepSeek's all-three specificity was only 87.5%. Even when it correctly identified a defect, it gave a wrong fix in 6 cases.

The model can point to the right line but propose the wrong repair. That's more dangerous than a plain false positive — you'll assume the problem is solved when it isn't.

Why This Matters

This paper isn't just about backtest auditing. It reveals a broader issue:

When LLMs act as audit/review tools, recall and calibration are independent dimensions; looking only at recall is misleading.

The conclusion applies to any "find problems" scenario:

• Code review: LLMs find bugs but also flag good code as buggy
• Security auditing: LLMs find vulnerabilities but also flag benign code as malicious
• Paper review: LLMs find methodological issues but also flag sound methods
• Compliance checks: LLMs find violations but also flag compliant behavior

In all these cases, counting found problems is not enough — you must also count false alarms, whose cost isn't just time but diluted attention to real issues.

The Clean-Aware Prompting Insight

The paper's most practical finding: adding one sentence — "clean strategies are common" — dropped DeepSeek's false positive rate from 20.8% to 0.0% while keeping recall intact.

The implication goes beyond finance: LLM judgments are influenced by prior probabilities. Tell it only "find problems" and it assumes every input has problems. Tell it "most inputs are clean" and it adjusts its threshold.

Human auditors behave the same way. Tell an auditor "this team has a history of issues" and they turn paranoid, flagging everything. Tell them "this team is consistently rigorous" and they allocate attention more rationally.

An LLM's "paranoia" isn't a bug — it's a prior baked into your prompt design.

Honest Assessment

The paper has limitations:

1. Only 8 defect categories. Real backtests have subtler problems: data leakage, overfitting, selective reporting.

2. Only 4 models, all lightweight. Stronger models (GPT-4, Claude Opus) may behave differently.

3. Simulated environment. Constructed backtest code; real strategy code may be messier.

4. Generalization of the clean-aware effect. It worked dramatically on DeepSeek but not others. Why? The paper doesn't dig deep.

Still, the contribution stands: it turns "LLM audit quality" from a one-dimensional problem (recall) into a two-dimensional one (recall + calibration), with a clean paired-benchmark paradigm for measuring both. Any system using LLMs to "find problems" should be evaluated similarly.

---

Paper: Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark

Authors: Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, Arsenii Bobovnikov (MBZUAI + University of Pittsburgh)

Bottom line: LLM auditors can easily hit 100% recall but struggle with calibration. On a paired clean-control benchmark, open prompts produced false positive rates as high as 93.8%. Clean-aware prompting reached 100% recall with 0% false positives — but only on one model. Reporting recall alone severely overestimates audit quality.