Summary
This post discusses the LiveSearch paper (arXiv:2610.37166) from Microsoft Research Asia and collaborators, which redesigns financial search-agent benchmarks so that gold answers are generated daily by executable programs rather than stored as frozen strings.
Key points
- Three diseases of static benchmarks share one root cause — storing questions as (text, answer) pairs:
- Curation is expensive; refreshing requires redoing manual annotation.
- Frozen answers silently go stale, and leakage rewards memorization over search.
- Questions drift toward free web-data conventions instead of professional database fields (adjusted prices, disclosure periods, official yield curves).
- The fix: a (text, π, τ) triplet.
text: question frozen with a hard rule banning absolute dates (only relative expressions like "latest trading day").π: an executable verifier — a typed DAG of tool calls plus a declarative extractor (no branching/expressions, so replay is deterministic and trivially verifiable) running on 9 data services with 96 tools over MCP.τ: matching rule (exact / approximate / semantic).- Daily re-execution writes dated snapshots
gold_answers_<date>.jsonlrecording formatted answer, raw value, observation period, and parameters. - Answer-first pipeline. LLMs only translate an already-fixed extractor into natural language — they may never touch which fields define the answer. Stage 2 (pool selection, daily refresh) is fully automated with no LLM involvement, cutting refresh cost to an unattended routine job.
- Measurement subtleties.
- Three clocks can disagree: benchmark date, request cutoff, provider-reported observation period. Of 109 questions, 93 saw their observation period advance; all 76 gold drifts occurred there. 17 questions advanced observation periods yet kept stable formatted answers — an unchanged answer is not evidence of refresh failure.
- Rendering matters: raw values changed on 77 questions but formatted answers on only 76 — a 0.039→0.038 HKD change rounds to 0.04 in all snapshots. Drift statistics depend on published precision, not just the underlying series.
- Headline results.
- Six models under pure web search scored 70.9%–76.7% (DeepSeek V4.1 Flash best at 76.7, Kimi K3 lowest at 70.9).
- Across Sept 14–16, 2026, 76/109 questions changed gold at least once (67 and 69 changed between adjacent days); yesterday's gold submitted as today's prediction failed 66/109 and 68/109.
- 4 answers lapsed then "revived" a day later as prices round-tripped — expiry is not monotonic, and snapshots capture such reversals.
- Per-question correctness flipped 19–22 times between days for three models while aggregate accuracy barely moved (0, −3, −2): leaderboard stability hides answer churn.
- In an 86-question pilot, 41 were solved by all, 10 by none; the 35 discriminative questions clustered in confusing financial conventions and edge microstructure cases.
- Failure modes: not insufficient search. Agents searched multiple rounds and cross-checked pages; errors came from convention mismatch — (1) differently-adjusted (e.g., back-adjusted) prices across portals, (2) ambiguity in "latest trading day" (zero-volume days), (3) unverified second-hand values (aggregator 1.247% vs official curve 1.2439%). The bottleneck is aligning data conventions, not finding pages.
- Implications.
- Benchmark designers: no absolute dates, regenerable answers, timestamped snapshots, published rendering precision — applicable to weather, sports scores, policy, drug guidance, etc.
- Model teams: memorization is increasingly worthless; only genuine daily search/compute wins.
- Product teams: "user's now," "provider's observation period," and "cache cutoff" are three different things.
- Live benchmarks double as contamination detectors: a model whose scores never fluctuate with the market is a memorization suspect.
- Sinuo Wang et al., "LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents," arXiv:2610.37166 [cs.AI], 2026.
- Zongyuan Sun et al., "Clearing the FoG: Training KG Agents to Retrieve the Evidence Behind Their Answers," arXiv:2610.37405 [cs.AI], 2026.
- Wadie El Amrani, Luis Galárraga, "GraphRAGProv: GraphRAG Meets How-Provenance," arXiv:2610.37237 [cs.AI], 2026.