Clearing the FoG: Training KG Agents to Retrieve the Evidence Behind Their Answers

*An annotated Chinese forum review of the paper "Clearing the FoG: Training KG Agents to Retrieve the Evidence Behind Their Answers" (arXiv:2610.37405), by Zongyuan Sun, Shuai Zhao, Ke Shu, Yingjun Du, Ziyue Zhu, Haoran Luo (Beijing University of Posts and Telecommunications, University of Helsinki, University of Amsterdam, Nanyang Technological University).*

Key points

The problem: correct answers without evidence

• Modern KGQA agents (e.g., over Wikidata) achieve strong accuracy but often reach answers via pretraining memory rather than the graph. On GTSQA, at least 9.2% of PoG's correct answers never appeared in any KG observation; for KG-R1, 45.95% of correct answers lacked retrieved supporting triples (Table 4).
• Two root causes in training signal:
1. Data: mainstream KGQA datasets annotate answers (at most logical forms) but not the supporting triples; retroactive annotation costs explode combinatorially with hop count. 2. Reward: RL rewards only check the final answer, making a lucky partial-evidence trajectory worth the same as a fully grounded one.
• Consequences: unauditable answers, encouraged hallucination (bypassing the graph), and reward hacking.

The three-part method

1. Evidence-Grounded Synthesis: sample Wikidata subgraphs, then use a teacher model (Kimi-K2) to generate questions whose logical forms and supporting triples ship together. Backward Expansion replaces explicit entities with sub-questions (matryoshka-style), yielding 1–3 hop questions; Proposition 1 proves the answer is preserved and accumulated triples still support it. Quality gates: executability (answer within a bounded result set) and non-redundancy (every topic entity is necessary; only 1.5% shortcut leakage in a 200-question human audit). 2. KG Interaction Agent: action space of think, kg-query (7 atomic graph operations), and answer; every trajectory logs all returned triples as T(τ). 3. Evidence-Coverage Process Reward (ECPR):

Cov(τ, S*q) = |T(τ) ∩ S*q| / |S*q|

Naive variants fail: trajectory-level reward loses 4.35 coverage; every-match reward inflates duplicate retrieval from 8.6% to 29.4%. First-match attribution (each gold triple rewarded only at its first retrieval) fixes this; Proposition 2 shows ECPR is maximized exactly when the answer is fully supported. LLM-as-judge ablations underperform ECPR by 4.22–5.28 coverage. Final reward: R(τ) = R_out + R_step + R_ecpr with λ_f1=0.7, λ_ret=0.3, λ_step=0.5, λ_ecpr=1.

Results

• FoG-4B tops all three benchmarks: WikiWebQuestions (F1 66.41, Hit@1 76.93), QALD10-en (F1 47.80, Hit@1 49.01), GTSQA (F1 48.66, Hit@1 50.52); up to +3.73 F1 / +3.92 Hit@1 over KG-R1. FoG-3B even beats the 4B KG-R1.
• Evidence metrics: RetQ 86.35 / 65.34 and Cov 78.34 / 58.64 (WikiWebQuestions / GTSQA), both best-in-class.
• Removing ECPR drops coverage by 5.43 / 7.21 — the reward is the load-bearing component. ECPR's gains grow with structural complexity (+1.4 on (1)-type up to +6.0 on (3)(1)-type questions).
• Training dynamics are healthy: reward rises while KG interaction turns and response length both fall — more evidence with fewer queries.
• Zero-shot cross-graph transfer: trained on Wikidata, FoG scores 77.2 Hit@1 on Freebase WebQSP, near natively-trained baselines (82.6–85.7), without any schema alignment.
• Synthetic-only training outperforms train-only by +3.66 / +2.50 / +3.06 Hit@1 — evidence-annotated synthetic data beats unannotated real data.

Limitations acknowledged by the authors

• Graph incompleteness/staleness; reliance on the Kimi-K2 teacher (self-play proposed as an alternative); a 10-round interaction cap.

Reviewer's takeaway

The paper's real contribution is a paradigm shift: move evidence annotation from "post-hoc archaeology" to "design-time embedding." The reviewer argues RetQ and Cov should be reported like nutrition labels for any KG agent, and notes this is part one of a planned series (GraphRAGProv, LiveSearch).

Reference

• Zongyuan Sun, Shuai Zhao, Ke Shu, Yingjun Du, Ziyue Zhu, Haoran Luo. "Clearing the FoG: Training KG Agents to Retrieve the Evidence Behind Their Answers." arXiv:2610.37405 [cs.AI], 2026.