FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents

Paper: FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents

Authors: Jia Deng, Yimeng Chen, Xiaoqing Xiang, et al. (Renmin University of China GSAI + KAUST + IQuest Research + Shanghai Jiao Tong University)

arXiv: https://arxiv.org/abs/2606.12087

Code: https://github.com/RUCAIBox/FORT-Searcher (to be open-sourced)

---

TL;DR

Existing deep-search datasets look complex (multi-hop, large evidence graphs, scattered clues), but large models almost always find shortcuts—guessing after one or two searches instead of learning long-horizon search. This paper is the first to systematically define this "shortcut problem" and proposes FORT, a framework that blocks four classes of shortcuts at the data-synthesis stage. FORT-Searcher, trained with pure SFT, achieves the best results among similarly sized open-source models on hard benchmarks like BrowseComp.

---

The Core Problem: Structural Complexity ≠ Real Difficulty

Consider a task like:

> "Find a botanist who described a fern named after a mountain, whose PhD advisor also supervised another botanist who discovered an orchid, served as president of a botanical society, and whose years in office sum to 7."

This looks like a genuine multi-hop deep-search task. But GPT-4 or Claude may guess the answer directly from memorized knowledge, or find the key clue in one or two searches because a single Wikipedia page happens to mention both the fern and the orchid.

The paper's central insight: preset task-structure complexity does not equal actual search difficulty at execution time. Existing synthesis methods (adding hops, expanding graphs, scattering evidence) only control "how hard it looks," not "how hard it is to solve."

---

Theoretical Framework: Four Shortcut Factors

The paper formalizes multi-constraint retrieval tasks as a triple q = (𝒳, 𝒞q, Σ):

• 𝒳: candidate answer space (e.g., all botanists)
• 𝒞q: set of constraints in the question
• Σ: retrieval interface (e.g., search engine)

Key insight: task difficulty is determined not by the most complex path, but by the cheapest "identifying route". Even in a huge evidence graph, if a small clue subset pins down the answer, the model takes that shortcut.

Four-Factor Difficulty Formula

Via formal derivation (Proposition 1), search cost is jointly determined by four factors:

| Factor | Symbol | Meaning | Prevents | |--------|--------|---------|----------| | Subset selectivity | s(𝒫) | Ability of a small clue subset to narrow candidates | "One clue determines the answer" | | Evidence dispersion | Mev(𝒫) | Minimum number of independent retrievals to verify clues | "One page covers many clues" | | Dependency depth | dep(𝒫) | Longest dependency chain among queries | "Constant exposure skips steps" | | Prior utility | Uπ0(q) | Search cost the model saves from memory | "Model guesses the answer" |

The first three are route-level shortcuts (shortening the cheapest identifying route); the last is a solver-level shortcut (favoring specific models).

Three Trajectory Signatures for Diagnosis

Since the four factors aren't directly computable, the paper proposes three observable trajectory signatures:

1. Solution cost Ω̂ — average retrieval steps in successful trajectories. High ≠ hard; it may just be wasted detours. 2. Answer hit time T̄hit — average step at which the answer first appears (in retrieved results or model outputs). The key metric: the later the hit, the better shortcuts are blocked. 3. Prior shortcut rate p̂prior — the fraction of times the model states the answer before anchoring it with retrieved evidence. High = memory-based cheating.

Diagnosis of existing datasets (Table 11) is sobering:

• OpenSeeker: Ω̂ = 84.7 (long trajectories), but T̄hit = 9.3 (answer appears at step 9), p̂prior = 31.9% (nearly a third from memory)
• REDSercher: relatively better trajectories, but answer hit time is still far earlier than total steps

→ Conclusion: long trajectories ≠ deep search; late answer appearance is the real depth.

---

FORT: Four-Stage Shortcut-Resistant Data Synthesis

FORT (Framework of Shortcut-Resistant Training-Data Synthesis) turns the theory into four engineering stages:

Stage 1: Graph Initialization

Goal: Reduce prior-knowledge binding + avoid early constant exposure
• Long-tail root entity selection: choose obscure, long-tail entities as roots instead of famous ones—the more obscure, the less likely the model already knows the answer.
• Cyclic initialization: use cyclic rather than linear-chain structures to avoid exposing needed intermediate constants in one step.

Stage 2: Graph Construction

Goal: Scatter evidence + weaken individual clues
• Multi-source enrichment: collect facts from heterogeneous sources (different sites, databases) so no single page covers multiple clues → counters evidence co-coverage.
• Derived fact construction: instead of using original statements, build derived facts requiring inference (e.g., "A and B collaborated" → "A and B co-published papers"), lowering single-page hit probability.
• Fact selection strategy: each clue is individually ambiguous (weak), but jointly uniquely identifies the answer (strong) → counters single-clue selectivity.

Stage 3: Question Formulation

Goal: Hide actionable constants
• Intermediate entity hiding: don't name key intermediate people/places; the model must discover them via search before issuing the next query.
• Precise value fuzzing: replace exact numbers/dates with ranges, categories, or indirect descriptions. E.g., "2023" → "early 2020s"; "3650 m altitude" → "a high mountain region above 3000 m."
→ Counters dependency-depth shortening from exposed constants.

Stage 4: Adversarial Refinement

Goal: Use strong models to "attack" weak samples
• Run each candidate question with a strong search agent (e.g., GPT-4 + search tools).
• Analyze its trajectory signatures: low Ω̂, early T̄hit, high p̂prior → the sample has a shortcut.
• Repair strategies:
• Shortcut samples: add constraints, disperse evidence sources, deepen dependency chains
• Ambiguous samples: clarify wording while keeping fuzzing strategies
• Over-fuzzed samples: fine-tune ranges to make them solvable

This is effectively a red-team test for the dataset—let the strongest model play hacker, find the holes, then patch them.

---

Results: SFT-Only Is Enough to Top the Charts

FORT-Searcher (pure supervised fine-tuning on FORT data) across hard deep-search benchmarks:

| Benchmark | Description | FORT-Searcher | |-----------|-------------|---------------| | BrowseComp | English deep search | Best among same-size open models | | BrowseComp-ZH | Chinese deep search | Best among same-size open models | | xbench-DeepSearch | Multi-domain deep search | Best among same-size open models | | Seal-0 | High-difficulty held-out set | Best among same-size open models |

Key comparison: FORT-Searcher does not use RL—SFT alone beats other open models (including RL-trained ones) on these long-horizon search benchmarks.

Data quality comparison:

• FORT tasks show significantly later answer hit time (T̄hit) than existing datasets
• Significantly lower prior shortcut rate (p̂prior)
• Successful trajectories concentrate in the "long search" regime

→ Conclusion: data quality (shortcut resistance) is itself the strongest hyperparameter.

---

Why This Paper Matters

1. From "structural engineering" to "adversarial engineering"

Previous data synthesis was forward design—make tasks as hard as you want by adding hops and dispersion. FORT is inverse verification—first define what a "shortcut" is, then attack your own design with strong models and patch until no holes remain. This mirrors red-team testing in cybersecurity and points to where AI training data is heading.

2. SFT beating RL

The most counterintuitive result: SFT-only achieves SOTA. This suggests that on many tasks the model isn't incapable of searching—it's that the training data never gave it a signal that "searching is required." If the data is full of "guess after two searches" samples, the model never learns to persist for 20 steps. FORT shows data quality can partially substitute for training-algorithm complexity.

3. A transferable "shortcut theory"

The four shortcut classes (co-coverage, single-clue selectivity, exposed constants, prior binding) aren't unique to search. Any multi-step reasoning/tool-use scenario—code generation, scientific computing, complex decision-making—faces similar issues. The framework can transfer to:

• Multi-hop QA dataset construction
• Tool-learning training data
• Complex instruction-following data
• Even human exam design (preventing shortcut cramming)

4. Practical engineering

Every FORT stage has operational definitions (how to select long-tail entities, how to do multi-source enrichment, fuzzing rules), unlike purely formal theory work. Highly relevant for industrial data-synthesis pipelines.

---

Limitations and Open Questions

1. Long-tail cold start: if the root entity is too obscure, search engines may not have enough content to build the evidence graph, raising synthesis failure rates. The paper doesn't report synthesis success rates. 2. Fuzzing boundary control: too little fuzzing leaves shortcuts; too much makes tasks unsolvable or ambiguous. Adversarial refinement fixes part of this, but automated control still needs manual tuning. 3. Complementarity with RL: only SFT is evaluated. FORT data + RL (e.g., R1-Searcher's method) could be stronger—a natural next step. 4. Multilingual/domain transfer: experiments focus on BrowseComp (English/Chinese). Whether FORT works for deep search in scientific literature, medicine, or law needs more validation.

---

References and Resources

• FORT-Searcher: https://arxiv.org/abs/2606.12087
• Code: https://github.com/RUCAIBox/FORT-Searcher
• Related data-synthesis work: REDSearcher (Chu et al., 2026), OpenSeeker (Xia et al., 2025), WebShaper (Tao et al., 2025), MiroThinker (Team, 2026)
• RL search training: R1-Searcher (Song et al., 2025), SmartSearcher (Song et al., 2025)
• Benchmarks: BrowseComp (OpenAI, 2025), xbench-DeepSearch (Zeng et al., 2026), Seal-0 (Tao et al., 2025)