The headline numbers
Same architecture, same hyperparameters, same data — only the random seed changed. The Sharpe ratio of an identical equity trading strategy swings between 0.233 and 0.855 across 20 runs, with a mean of 0.604. Reporting only the best seed inflates the Sharpe by 44%.
These figures come from a secondary summary (Tech Daily Byte) of the paper; the poster could not locate the exact table in the paper itself, so treat the paper as authoritative.
The paper
Unstable Gains: Multiplicity-Aware Evaluation of Financial Deep Reinforcement Learning
- Journal: The Journal of Finance and Data Science, vol. 12, article 100205
- DOI: 10.1016/j.jfds.2026.100205
- Open access, published 2026-09-23
- Best seed: 132.6% out-of-sample total return, Sharpe 1.78
- Five-seed mean Sharpe: 0.79 — below all classical baselines (equal-weight, buy-and-hold, momentum, minimum variance)
- The most successful runs came with significantly higher turnover and portfolio concentration: returns came from aggressive reallocation and risk concentration, not stable risk-adjusted efficiency
- Run every strategy across multiple seeds; report the distribution and the number of runs, not the maximum
- Pass algorithm comparisons through permutation tests and family-wise error correction before reading any leaderboard
- Always report Sharpe alongside the Deflated Sharpe Ratio, especially for high-volatility markets
- Maintain a trial log: feature sets, model types, label definitions, parameter sweeps — every change counts as a trial
- Split data by date, not randomly; apply purge and embargo to overlapping labels
- Fix the cost model up front; examine cost break-even points separately for high-turnover strategies
- *Unstable Gains: Multiplicity-Aware Evaluation of Financial Deep Reinforcement Learning*, The Journal of Finance and Data Science, vol. 12, article 100205 (2026), DOI 10.1016/j.jfds.2026.100205, open access
- Bailey, D. H. & López de Prado, M., *The Deflated Sharpe Ratio*, SSRN 2460551 (2014)
- *Seed Sensitivity and Risk Efficiency in PPO-Based Portfolio Allocation*, Research Square preprint rs-9824202, not peer-reviewed
- Bailey, Borwein, López de Prado, Zhu, *Pseudo-Mathematics and Financial Charlatanism*, Notices of the AMS (2014)
- Tech Daily Byte's aggregated summary of three 2026 studies (0.233–0.855 range, 0.604 mean, 44% inflation, plus two untraceable papers)
The question is simple: how should we evaluate reliability of DRL financial decision-making under non-convex stochastic optimization? The authors identify two pervasive practices — single-run backtests and algorithm comparisons without multiplicity correction — that create an illusion of consistently superior economic performance.
The experiments fix hyperparameters and retrain existing DRL architectures across multiple independent runs on two task types: equity trading and cryptocurrency portfolio allocation.
Three findings
1. Seed sensitivity. Sharpe ratios differ significantly across random seeds. Apparently superior performance often disappears under multi-run, multiplicity-aware evaluation. 2. Leaderboard comparisons collapse. Algorithm rankings that look economically meaningful lose statistical significance after permutation tests and family-wise error correction. This directly undermines the industry habit of ranking algorithms in a table. 3. Deflated Sharpe Ratio matters. In high-volatility crypto markets, Sharpe ratios of the same magnitude can receive completely different diagnostic support under the Deflated Sharpe Ratio (Bailey & López de Prado, 2014), which corrects for selection bias, backtest overfitting, and non-normality.
> The evidential value of an anomalous backtest depends not only on its magnitude but on how rare it is relative to the distribution of results from random training runs.
Core methodological claim
When a trading strategy is produced by non-convex stochastic optimization, performance should be treated as a random variable over the learning procedure, not a deterministic function of the data. A strategy should be reported as a distribution, not a single number. The proposed framework rests on three pillars: multi-run evaluation, uncertainty quantification, and explicit multiplicity control.
> Without multiplicity-aware evaluation and stability analysis, reported improvements in financial DRL may reflect the stochastic optimization and model selection process itself, rather than robust economic performance gains.
Independent diagnosis: RiskLens Trader (PPO)
A companion preprint on Research Square (rs-9824202, not peer-reviewed) tests PPO on five US large caps (AAPL, MSFT, NVDA, AMZN, GOOGL), 2018–2026, chronological 80/20 train/test split, five random seeds, everything else fixed.
Summary table
| Material | Object | Key numbers | Status | |---|---|---|---| | Unstable Gains | DRL for equity trading & crypto allocation | Wide cross-seed Sharpe dispersion; leaderboard comparisons often lose significance after correction | Journal publication 2026-09-23, open access | | RiskLens Trader | PPO, 5-stock portfolio | Best seed Sharpe 1.78; mean 0.79 below classical baselines | Preprint, not peer-reviewed | | Deflated Sharpe Ratio | General statistic | Downward-corrects Sharpe by number of trials, sample length, non-normality | Bailey & López de Prado 2014, SSRN 2460551 |
Two additional claims from the secondary source (volatility forecasting models with near-identical out-of-sample error but up to 3x turnover differences; backtests appearing significant on synthetic markets with zero predictability) could not be traced to primary sources and are listed as leads only.
Engineering checklist
Decision rule for a single-run backtest conclusion: Was the number of runs reported? No → not credible. Was it tested with permutation tests and family-wise correction? No → not credible. Was Sharpe corrected for the number of trials? No → magnitude plausible, evidence strength undetermined. Yes on all three → usable as decision input.
Takeaway
The paper does not prove DRL is ineffective for trading. It proves something else: under current reporting conventions, we cannot tell whether a good result reflects a good model or good luck.
Three things to watch: whether the proposed reporting framework becomes a standard in follow-up empirical work; whether journals and conferences make multi-run reporting a review requirement; and whether quant teams are willing to publish their own trial logs alongside results.