The Devil's Advocate: A Search Engine That Learned to Tear Itself Down

*On FALCA: falsification-aware reinforcement learning for long-horizon deep search agents*

Opening: A Detective Who Never Suspected His Own Suspect

The post opens with a parable: a detective locks onto the wrong suspect on night one, then spends three months collecting only corroborating evidence. Contradictory details get buried under fresh confirmations. The harder he works, the more solidly wrong he becomes — search itself becomes a fermentation machine for error.

Real criminal justice systems solved this structurally, not by making detectives smarter: dedicated devil's advocates (defense attorneys, internal affairs, peer review). Popper elevated this to philosophy: a theory that cannot be falsified does not deserve to be called science.

Deep search agents — OpenAI Deep Research and similar systems — scale up the detective. They are much better at *finding*, not at being *right*. And the deeper the search, the more the flaw is amplified, not diluted.

This is the third and final post in a "triple distrust" series: prior posts covered over-trust of user assertions and of adversarial arguments. This one covers the hardest case: an AI trusting its own conclusions.

Two Ways to Die, One Loop

The paper (BAAI) dissects two failure modes of deep search agents:

• Confirmation bias: the agent anchors on a candidate early; subsequent searches become one-way resupply for that anchor rather than open exploration.
• Error accumulation: a wrong candidate entering the research context pollutes the notes all later reasoning is built on. A wrong name ingested at round three has become a load-bearing wall by round ten.

The sharp diagnosis is that these feed each other in a self-reinforcing loop: confirmation bias hides errors better; accumulated errors make the anchor look more credible. Traditional fixes fail: stronger models anchor more confidently; larger context gives pollution more room; more search rounds are literally the loop's fuel.

The cure must come from structure.

Propose–Falsify: Writing Popper into the Search Loop

FALCA redefines deep search as a propose-falsify process: propose candidate → adversarial challenge → revise research state → propose again. Falsification becomes an *in-process context management mechanism*, not post-hoc verification. Crucially, self-critique is not enough — the same model on the same polluted context is like a detective defending himself. Skepticism must be structurally separate: a different agent, seeing original evidence, not the proposer's reasoning.

Paper: *FALCA: Falsification-Aware Reinforcement Learning with Hierarchical Credit Assignment for Long-Horizon Deep Search Agents*, Zijun Dou, Kun Luo, Chaofan Li, Shuqi Lu, Zheng Liu (BAAI, Tsinghua University, CASIA), arXiv:2610.28361, 2026-10-20.

Architecture: One to Think, One to Break

Two roles share a backbone (Qwen3.5-4B), but with fully separated contexts:

• Searcher: decomposes the problem, collects evidence, maintains research state, proposes candidates. Tools: Search, Visit, Update Context, Finish, and Refute — the only tool that reaches outside, making challenge a native in-flow action.
• Falsifier: lives in an isolated context, seeing only the candidate hypothesis and its evidence. It runs contradiction-first adversarial stress tests — finding counter-evidence first is efficient, because a candidate's death sentence needs one hard contradiction, while praise letters never deliver one. Its output is a structured report: verdict (reject / survive / uncertain), refuted constraints, unresolved constraints, evidence, recommended queries.

Rejected candidates are removed; unresolved constraints become new search targets — failures are dissected into navigation. Notably, the Falsifier never supplies answers, only challenges — if it did, the two would conspire.

Training: Learn the Craft, Then Work Independently

1. SFT warm-up on falsification-guided trajectories (tools, context updates, coordination, candidate revision). 2. Falsification-aware agentic RL with four components:

• Parallel rollouts (n trajectories per question).
• Asynchronous execution.
• Hierarchical reward: trajectory-level (final-answer exact match + LLM judging), segment-level (intermediate subgoal/entity discovery, aligned with each context-update boundary), turn-level (LLM judge scores -1 to 2 per action).
• Composite advantage:

A_i = λ_traj · R_traj/√K + λ_seg · R_seg + λ_turn · R_turn

Optimizer: PPO with DAPO-style asymmetric clipping plus token-level KL regularization to the SFT reference. Evaluation/summary during falsification uses Qwen3.5-35B-A3B; retrieval uses local HNSW + BGE-m3; training on 8× H100 80GB. Only the 4B Searcher is trained — the 35B sits on the judge's bench.

Why Not GRPO: Two-Thirds Dead Heat

About 66% of rollout groups have zero reward variance — all-correct or all-wrong, since long-horizon deep search is hard and paths converge. GRPO's group-relative advantage then evaporates exactly where learning is needed. The hierarchical absolute advantage keeps learning signal alive even in degenerate groups: training curves show GRPO-style optimization plateauing early while the hierarchical advantage keeps climbing. Practical lesson: before choosing an RL algorithm for agents, measure how many of your groups are ties.

Scoreboard: 4B Body, 30B Brain

| Model | Average | |---|---| | Qwen3.5-4B (default harness) | 51.8 | | + Falsifier harness (no training) | 55.5 | | FALCA-4B | 64.7 | | Qwen3.5-9B (same harness) | 57.9 | | Qwen3.5-397B | 74.0 |

Read top-down: the structure alone is worth +3.7; full SFT + RL adds more; FALCA-4B beats the 9B by 6.8 despite half the parameters (biggest gap on BrowseComp: 56.0 vs 41.4 — exactly where deep digging multiplies wrong candidates). Against the 397B giant it trails by 9.3, but reaches 30B-class specialized-agent territory at 4B inference cost. For product builders: much of the deep-search capability gap depends on structure, not model size.

Autopsy: Remove the Falsifier, Everything Collapses

| Variant | Average | |---|---| | Full FALCA | 64.7 | | Without Falsifier | 52.8 | | Without agentic RL (SFT only) | 60.5 | | Without process reward (trajectory only) | 60.1 |

Removing the Falsifier costs 11.9 points — nearly three times worse than removing all RL — collapsing back to near the default harness. Adversarial challenge is the load-bearing wall.

Behavior: A Real Case Overturned, a Counterintuitive Scaling Law

On xbench-2510, the Searcher proposed Alibaba; the Falsifier rejected it on three checkable violations (listing year, CFO's investment-banking background, appointment date). The Searcher switched to VNET, verified all constraints, and answered correctly. Also notable: the Searcher obeys the Falsifier's verdicts — the veto power is wired into the state-revision interface, which is why the adversarial structure actually works.

Long-horizon behavior: 50% of correct trajectories finish within 20 rounds, 89% within 120, only 93% even at 200. Rounds are not the bottleneck; quality of challenge is. Scaling rounds, parameters, or brute force all hit diminishing returns against the error loop; only structurally injected skepticism compounds.

Epilogue: The Third Distrust

The series completes an arc: distrust users (they err), distrust rhetoric (it manipulates), distrust yourself (you anchor). FALCA institutionalizes self-doubt — in tool definitions and reward functions — with the moderation of three verdicts: reject, survive, uncertain. The recipe extends beyond search: any long-horizon, multi-step agent workflow (coding, research, planning) breeds the same disease of early errors polluting later context. A system that cannot tear itself down will collapse all the more spectacularly as it scales.

---

Reference: FALCA: Falsification-Aware Reinforcement Learning with Hierarchical Credit Assignment for Long-Horizon Deep Search Agents — Zijun Dou, Kun Luo, Chaofan Li, Shuqi Lu, Zheng Liu (BAAI, Tsinghua University, CASIA) — arXiv:2610.28361, 2026-10-20