QUEST: Open-Source Deep Research Agents Trained on Just 8,000 Synthetic Tasks, Rivaling Closed-Source Systems

OSU NLP Group, in collaboration with Amazon AGI SF Lab, has released QUEST — a fully open-source family of deep research agents spanning 2B to 35B parameters. Across 8 diverse benchmarks, QUEST approaches or even exceeds closed-source frontier systems such as OpenAI DeepResearch and Kimi Researcher.

The headline number: it was trained with only 8,000 synthetic tasks.

Why This Matters: The Closed-Source Wall Is Being Pushed Back

Deep research agents are among the hottest areas of 2025–2026. Systems like OpenAI DeepResearch, Kimi Researcher, Manus Wide Research, and Anthropic's multi-agent research have advanced search engines from "returning keyword-matched pages" to "autonomously investigating and generating fully cited reports."

But all of them are closed-source: no model weights, no training data, no training recipes.

Open-source efforts are catching up — Tongyi-DR, OpenResearcher, SFR-DeepResearch, OpenSeeker — but each covers only specific scenarios. None offers a unified, strong solution across task types.

QUEST aims to fill this gap: a general-purpose, fully open-source deep research agent that is strong across task types.

Core Innovation 1: The Unified Rubric Tree — Structuring "What a Good Answer Means"

Deep research evaluation naturally splits into two modes:

• Objective tasks: externally verifiable answers with clear correctness (BrowseComp, GAIA)
• Open-ended tasks: multi-dimensional subjective judgments (DeepResearch Bench — coverage, organization, clarity, insight)

Existing open agents mostly train on "complex question + single verifiable answer" data, which generalizes poorly and provides only sparse, binary rewards for RL.

QUEST's solution: the Rubric Tree — not merely an evaluation tool but the meta-structure of the entire training framework. It is a hierarchical decomposition of constraints:

• Root node: final total score
• Internal nodes: high-level constraints (e.g., "technical comparative analysis"), recursively decomposed
• Leaf nodes: directly verifiable fine-grained criteria (e.g., "cited Apple's foldable-screen patent"), each scored binary

For objective tasks (e.g., "which of two 2024 US listeria outbreaks killed more people"), the tree decomposes into independently verifiable nodes. For open-ended tasks (e.g., "evaluate Apple entering the foldable phone market"), the root has four fixed shared criteria — Instruction Following, Comprehensiveness, Readability, Insight — with adaptive sub-nodes.

This design delivers:

• A unified framework covering both objective and open-ended tasks
• Fine-grained rewards: partial scores at the root give RL richer signals than binary right/wrong
• Automatic scalability: GPT-5 translates rubric trees into Python verification code, no manual scripts needed

Core Innovation 2: A Fully Synthetic Data Pipeline — Where 8,000 Tasks Come From

QUEST's training set, QUEST-8K, is entirely synthetic with zero human annotation.

Objective task synthesis: trending keywords from Google Trends seed topics; Claude Sonnet 4.5 browses the web, collects information, and extracts verifiable constraints; these are organized into rubric trees, iteratively refined and self-validated; GPT-5 then generates Python evaluation scripts.

Open-ended task synthesis: the same keyword-driven exploration, but the rubric tree's root has four fixed shared criteria; sub-nodes are adaptively generated with weights averaged over three GPT-5 assignments; Claude Sonnet 4.5 produces reference reports.

Evaluation uses pairwise normalization: the judge scores both candidate and reference reports from 0–10, with final score = J(candidate) / (J(candidate) + J(reference)). Above 0.5 means the candidate beats the reference — avoiding score-scale drift across tasks of different difficulty.

Core Innovation 3: A Three-Stage Training Recipe

QUEST's training is not a one-shot SFT or RL run, but three sequential stages:

1. Mid-Training (MT): adapts base models (Qwen3-30B-A3B / Qwen3-35B-A3B) to the format and rhythm of long-horizon search interactions (search → read → reason → search again). 2. Supervised Fine-Tuning (SFT): teaches the agent to imitate high-quality tool-use trajectories. Notably, control experiments show SFT-only small models (2B–35B) are surprisingly strong on factual retrieval — QUEST-2B-SFT scores 30.3 on HLE and 72.8 on GAIA — meaning much deep research capability can be injected through imitation alone. 3. Reinforcement Learning (RL): optimizes the policy with rubric-based partial scores, letting the model learn trade-offs among multi-dimensional constraints and push beyond the imitation ceiling.

Ablations confirm each stage contributes: vanilla → +SFT (large gains) → +MT (better long-horizon interaction) → +RL (closing in on closed-source systems).

Core Innovation 4: Context Management — A Cognitive State Machine

A central challenge for deep research agents is long-horizon information loss: after ~20 search rounds, earlier findings get buried. QUEST's Context Management is a structured cognitive state machine:

A JSON Context State buckets accumulated knowledge into:

1. Trusted: facts verified via source URLs — reusable without re-verification 2. Untrusted: claims contradicted by other sources, with reasons for distrust 3. Uncertain: partially supported claims, each annotated with URLs to visit or queries to re-run

When the context window exceeds a threshold, a Context Condenser (GPT-5-mini) updates the Context State from the full raw history, and the agent continues in a fresh context window carrying only the structured state as "memory."

• Uncertain entries directly guide follow-up actions
• Trusted entries avoid redundant API calls
• Untrusted entries prevent circular re-verification

On BrowseComp and BrowseComp-Plus, QUEST used a "discard-all" strategy — relying entirely on the Context State for cognitive coherence rather than a giant context window.

Results: A Panorama Across 8 Benchmarks

| Benchmark | Type | QUEST-35B | vs. Closed-Source | |------|------|-----------|------------------| | BrowseComp | Factual retrieval | 64.6% | Near OpenAI DeepResearch | | Mind2Web 2 | Citation verification | 30.7% | Standout | | DeepResearch Bench | Report synthesis | 48.2% | Near or above closed-source |

• 30B-scale controlled comparison: Tongyi-DR is strong on factual retrieval (consistent with its single-answer training data); OpenResearcher leads on BrowseComp-Plus; QUEST-30B is best on Mind2Web 2 and DeepResearch Bench — cross-benchmark balance comes from the recipe, not just parameter count.
• Scaling trend: 2B → 4B → 9B → 35B shows monotonic improvement across all benchmarks, validating the recipe's scalability.

Why Only 8,000 Tasks Suffice

The paper's most counterintuitive number. While not explicitly explained, plausible reasons include:

1. High-density supervision: each rubric tree contains 10–30 nodes; 8,000 tasks yield hundreds of thousands of fine-grained training signals. 2. Inherently complex tasks: trajectories can span 10–50 steps, amplifying effective data volume. 3. Quality over quantity: strict self-verification and iterative refinement keep only high-quality samples.

This aligns with a broader trend: synthetic data may follow different scaling laws, where structural design and verification let small datasets produce outsized training effects.

What's Open-Sourced

• Models: 2B/4B/9B/30B/35B, each with MT, SFT, MT+SFT, and RL checkpoints
• Data: RL data, SFT objective and open-ended data, all public
• Code: inference pipeline, evaluation scripts, training scripts (SFT via LlamaFactory, RL via VERL + Megatron fully-async), and data generation pipeline
• Config templates: api_config.yaml, server_endpoints.conf, eval LLM configs

Limitations and Open Questions

1. Cached databases and mid-training data pending legal review — affects full reproducibility, especially for RL training that requires pre-built search/scholar/visit databases. 2. Open-ended tasks remain hard for small models: 2B–4B models struggle at report synthesis; long-form generation seems to have a hard scale requirement. 3. Judge model bias: open-ended evaluation relies on Claude/GPT-5 as judges, inheriting their biases. 4. Condenser single point of failure: long-horizon coherence depends on GPT-5-mini's condensation quality.

One-Sentence Takeaway

QUEST unifies objective and open-ended training via rubric trees, transforms base models through a three-stage recipe, solves long-horizon memory loss with structured context management, matches closed-source frontiers on 8 benchmarks with only 8,000 synthetic tasks — and open-sources everything.

It demonstrates that the training recipe for deep research agents matters more than model scale.

References:

• Xie et al. (2026). QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks. arXiv:2605.24218.
• Project: https://osu-nlp-group.github.io/QUEST
• Code: https://github.com/OSU-NLP-Group/QUEST
• Models & Data: https://huggingface.co/collections/osunlp/quest