Overview
On October 1 at 07:19, Shuai Wu, Xue Li, Zhijun Wang and four co-authors submitted a computational audit (arXiv:2610.01213, a 45.8 MB file). Their question: can the ranking an LLM produces over one hundred stocks be turned into a tradable rule?
The answer: the overall shape survives; the individual entry/exit decisions do not.
How the audit was stripped down
The audit skeleton is four models plus five numerical comparison methods, all sharing one portfolio engine, run over 72 monthly holding periods covering the SSE 50, CSI 300, and CSI 500 index pools. Rankings use nine features. SSE 50 was run five times to measure variance under identical inputs.
The stripping method: fit model preferences into a linear rule over a development period, freeze the rule, then run three groups of tests — unseen months, larger pools, and controlled interventions (5,184 responses).
Three numbers side by side
| Metric | Value | Meaning | |---|---|---| | Spearman agreement (SSE 50) | 0.923 to 0.984 | Overall ranking nearly identical | | Agreement after transfer | 0.795 to 0.985 | Still close after switching pools | | Jaccard on entry/exit events | 0.000 to 0.125 | Specific buys/sells almost never match |
The gap between the first and third rows is the whole paper. A Jaccard of 1.0 would mean every entry/exit matches exactly; 0.125 means one match in eight.
> The paper's own wording: predicting individual entry/exit remains weak. Overall ranking can be approximated by compact rules; return conclusions depend on comparison family, uncertainty method, and tie handling.
The returns half
On historical means, model-based compound annual growth rates range from 6.26% to 11.17%. Respectable numbers.
The transaction-cost column is less flattering. At 10 bps per side with six-month blocks, twelve model-minus-rule comparison families showed no significant findings after factor controls and correction.
Three high-cost, twelve-month comparisons favored a rule called Terra in their twelve-test slices — but placed back in the full 144-cell sensitivity grid, results remained non-significant after correction.
The two input-intervention batches totaled 5,184 responses. The three-model batch produced no bootstrap-corrected findings at the main block length; the Luna row showed a row-order effect under HAC correction within its three-test family, but it vanished in the merged 24-item correction.
> Judgment: the row-order effect is the only statistic worth singling out. Different row orders produce different rankings, and this was significant within its own family. This is not return leakage — it is model sensitivity to input format, a different class of problem.
How it fits with two other recent papers
| Date | Paper | Level addressed | |---|---|---| | 09-28 | Backtest validation failure — RiskLens PPO, five seeds, DSR | Whether the validation mechanism itself deceives | | 09-29 | Price of Thought — 800,000 predictions bought less than a quarter basis point | Inference compute fails to buy net returns | | 10-01 | This audit — LLM rankings as testable economic rules | Whether rankings reduce to rules |
The shared undertone: the backtest curve is determined by the denominator, and the denominator is repeatedly underestimated.
A number you can compute yourself: in the 144-cell sensitivity grid, three high-cost comparisons each favored Terra in their twelve-item slices. Twelve out of 144 is 8% — the uncorrected hit rate, which goes to zero after Holm correction.
Four limitations that must be flagged
- arXiv v1 preprint, 45.8 MB, not peer reviewed
- The 72-month history spans roughly mid-2019 to mid-2025; market structure is not constant
- 10 bps single-side cost is an assumed value, treated as the primary specification
- Terra and Luna are codenames for rules in the paper; full rule definitions are not shown
- arXiv:2610.01213, Shuai Wu et al., seven authors, submitted 2026-10-01
- ML-Quant 2026-10-02, issue 133, quant research literature screening
- arXiv:2610.01325, PPO-HRAP, single-window single-asset baseline, submitted 2026-10-01
Inference
The methodological value here exceeds the value of the conclusions. Four models and five baselines on a shared engine, three post-freeze test groups, a 144-cell sensitivity grid, two multiple-correction families — this is a systematic closing of the common loopholes in LLM-ranking papers. Other teams working on similar questions can borrow the template directly.
The practical trading implication is negative: aggregate preferences are stable while stock-level decisions are noise, so the usable portion reduces to coarse industry or style exposure — a layer that does not require four language models to rank.