Large language model agents for portfolio management typically learn from experience by crediting or debiting each retrieved memory according to how much the decisions that used it earned. The flaw: every decision on the same day faces the same market. When the market rises, every retrieved experience gets credit; when it falls, every one gets debited. The accumulated scores in memory effectively rank market conditions, not experience quality.
Researchers at the University of Technology Sydney (Guanghao Wu, Zhuo Cai, Shoujin Wang) submitted a paper to arXiv on October 8 that runs controlled experiments on this credit scheme, calls it outcome credit, and measures how far it is from what it should actually capture.
The diagnostic table
On three real benchmarks, the authors compute two numbers for each retrieved experience: *outcome credit* is the average utility of drafts that used it; *contribution* is the average difference between drafts that used it and drafts that did not. The reference is the equal-weight 1/N portfolio over the same holding period, serving as a market proxy.
| Benchmark | outcome credit vs. 1/N | contribution vs. 1/N | credit vs. contribution | Days with detectable effect | |---|---|---|---|---| | PortBench-Full | +0.91 ± 0.00 | +0.04 ± 0.03 | +0.11 ± 0.06 | 3–4 / 56 | | PortBench-Raw | +0.92 ± 0.00 | +0.03 ± 0.04 | +0.04 ± 0.03 | 2–4 / 56 | | InvestorBench | +0.95 ± 0.00 | −0.07 ± 0.02 | −0.03 ± 0.02 | 126–127 / 149 | | ClassAlloc | +0.97 ± 0.00 | +0.02 ± 0.07 | +0.02 ± 0.07 | 53–57 / 71 |
All values are Spearman rank correlations. The pattern is clean: outcome credit correlates with market conditions at 0.91–0.97, while contribution is near zero (even −0.07). Credit and contribution correlate only at 0.04–0.11 — essentially unrelated.
The paper gives an algebraic explanation: since each experience appears in exactly half the drafts, the average utility of drafts using experience *j* equals the average utility of all eight drafts that day plus c_{t,j}/2 — and the first term tracks the market. Accumulating outcome credit therefore ranks experiences by the market conditions at the times they happened to be retrieved.
On PortBench, days with detectable effects number 2–4 out of 56; the paper notes this is "roughly what pure chance would produce." On InvestorBench, 126–127 of 149 days show detectable effects — 85% of days, memory genuinely changes decisions. The same credit rule is almost pure noise on one benchmark and almost pure signal on another.
Putting memory on trial
To ask "what did this experience change?", the method runs a controlled experiment: same decision, same day, same market, one draft with the experience and one without; shared components cancel in the difference.
Step one: eight drafts instead of sixteen. The retriever returns k = 4 experiences. Full enumeration requires 2⁴ = 16 drafts to compute marginal effects, and each draft is a paid LLM call that varies due to positive temperature. The authors use a fractional factorial design taking half: only subsets of even size — a resolution IV 2³ design. It includes the empty and full sets; each experience appears in four drafts and each pair in two, so single-experience effects are not confounded with two-way interactions.
Step two: Banzhaf values as contribution. Contribution is the average utility of drafts containing an experience minus the average of drafts without it — the design's main effect. The authors note that over all 16 subsets this is the standard Banzhaf value; in the half design it is aliased with interactions among the other three experiences, adding 13–17% to the noise variance versus the full 16-draft benchmark. Citing Banzhaf's 1964 work, they argue this weighting makes the Banzhaf value the most robust semivalue under noisy utility.
Anchors, and why they matter
Contributions are measured on a single day that never returns, and each draft is a noisy LLM sample. So single-day contributions may not generalize. The paper pools them across dates and similar experiences using a hierarchical Bayesian model (borrowing from small-area estimation), then applies a trust test in the spirit of prequential analysis: the agent acts on these values only when they are significantly positive on unseen dates. Otherwise it falls back to an anchor: stay near 1/N and learn only how far to deviate from the drafts.
The authors' stance on the 1/N reference is notable: they cite DeMiguel et al. (2009), which found no evaluated method consistently beat the equal-weight portfolio out of sample — the justification for shrinking toward a conservative target. MemTrial's anchoring mechanism is a reconstruction of this tradition for LLM agents.
Results, and one unfavorable number
Across four benchmarks against 15 methods, MemTrial's utility exceeds the best experience-learning agent by 21.2% (averaged over five settings), and it is the best of the 15 methods on the semi-synthetic PlantedMem environment where experience quality is known.
On PlantedMem, where experience carries real signal, the gain is 0.10 percentage points per month. On PortBench and InvestorBench, no LLM method beats 1/N; MemTrial is the only LLM method close to it (at most 2.2% below), while the best existing experience-learning agent lags by 15–38%.
The authors disclose several limitations: on PortBench-Full, the lead of 1/N and MemTrial depends on the final test month (November 2024), when crypto rose 129%; removing it, Similarity retrieval (top-4) and Hedge slightly lead. On ClassAlloc, a memory-free LLM beats 1/N and no memory variant surpasses it; the authors attribute this advantage to months before the LLM's knowledge cutoff (June 2024). PlantedMem is semi-synthetic, with known quality as a precondition. Utility is measured as certainty-equivalent (CEQ) return under investor utility, with a penalty of half the daily return variance times holding days; the risk-aversion coefficient γ is part of the investor profile θ = (M, m, γ).
The paper was published on October 8 in cs.AI under a CC BY 4.0 code license. The authors state the contribution lies not in the Banzhaf value itself but in how it is measured and used.
--- Source: arXiv:2610.11732v1, "MemTrial: Learning When to Trust Memory in LLM Portfolio Agents", submitted 2026-10-08 11:27:49 UTC, University of Technology Sydney, cs.AI, CC BY 4.0. Correlation coefficients and day counts in the first table come from Table 1; the 15-method comparison and PlantedMem ranking from Table 3; design details (resolution IV fractional factorial, 13–17% noise variance) from Section 3.2 and Appendix C.1; the trust test and anchor from Sections 3.4–3.5; per-benchmark failure cases from Section 4.2 and Appendices D.6 and D.11. Preprint, not peer-reviewed.