Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents in Quant Factor Research

Forum topic · 小凯 · 2026-09-25

Summary

This paper proposes a governed self-evolution framework for language-model agents running quantitative factor research. Agents may propose investment factors, but a frozen statistical referee—untouchable by the agent—must judge them. The referee scores candidates only on market outcomes revealed after submission, using a betting-based anytime-valid test, so its false-discovery guarantee holds at every stopping time under any proposal policy. Experiments cross three proposers (a script, a bandit, and a language model) with the frozen referee and three deliberately leaky referees, on a synthetic world with planted truth, a probe-authoring environment, and a ten-year walk-forward on the CSI 500. The frozen referee admits 5-11x fewer below-threshold factors than leaky referees, and no proposer can close the gap. The LLM proposer outperforms the script, matches the bandit, and can author its own diagnostic probes. The cost of certification is time: an admitted true factor waits roughly 500 trading days, so certified portfolios trail uncertified ones in Sharpe ratio.

Paper Overview

Field: Machine Learning Authors: Bo Qu, Mingguang Chen, Licheng Wang Published: 2026-09-25 arXiv: 2609.27051

Summary

Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retire them. The authors ask which of those jobs an agent should keep. Their answer is governed self-evolution: the agent may propose, and a frozen statistical referee that the agent cannot touch must judge.

The referee scores each candidate only on market outcomes revealed after submission, by betting, so its false-discovery guarantee holds at every stopping time for any proposal policy.

Experimental Setup

The authors cross three proposers (a script, a bandit, and a language model) with this referee and with three deliberately leaky ones, in three settings:

A synthetic world with planted truth
A probe-authoring environment
A ten-year walk-forward on the CSI 500

Key Findings

Who judges sets the number of false admissions: the frozen referee admits 5-11x fewer below-threshold factors than leaky referees (with the script proposer), and no proposer can close this gap.
Who proposes sets output: the language model outperforms the script, matches the bandit, and adds a capability the bandit lacks — authoring its own diagnostic probes.
Certificates cost time: an admitted true factor requires roughly 500 trading days of waiting, so certified portfolios lag behind un-gated ones in Sharpe ratio.

Takeaway

Judging belongs to the process; proposing and tool-making belong to the agent.

Original Abstract (excerpt)

> Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retire them. We ask which of those jobs an agent should keep. Our answer is governed self-evolution: the agent may propose, and a frozen statistical referee that the agent cannot touch must judge. The referee scores each candidate only on market outcomes revealed after submission, by betting, so its false-discovery guarantee holds at every stopping time for any proposal policy...

Paper link: arXiv:2609.27051

Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents in Quant Factor Research

Summary

Paper Overview

Summary

Experimental Setup

Key Findings

Takeaway

Original Abstract (excerpt)

Tags