DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research
- Paper: DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research (arXiv, May 2025)
- Authors: João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, et al. (11 authors)
- Category: Evaluation of Search Engines / Deep Research Agents
- Motivation: Deep research systems (LLM agents that search, browse, and synthesize cited reports) are typically evaluated against live commercial search APIs. This makes evaluation expensive, sensitive to opaque API-side changes, and effectively non-reproducible—different runs or labs may see different retrieval results for identical queries.
- Proposal: DeepResearchGym provides a free, open, and transparent alternative: a large-scale corpus paired with an open search API, so that retrieval-grounded evaluation is deterministic and reproducible at zero API cost.
- Benchmark: The work introduces DeepResearchBench, a suite of research tasks enabling controlled comparison of deep research agents under identical retrieval conditions.
- Decomposed evaluation: By controlling the retrieval layer, the sandbox separates the quality of the search/index stack from the agent's planning, querying, and synthesis behavior—allowing components (retriever, ranker, planner, generator) to be attributed credit independently.
- Evaluation should move beyond static ranking metrics toward task success, citation accuracy, and multi-hop reasoning quality, measured under reproducible conditions.
- LLM-as-judge signals need cross-validation with human assessment; controlled sandboxes make such audits cheaper.
- Engineering constraints—latency, cost, hallucination, and safety—remain decisive for production; a free sandbox lowers the barrier to stress-testing these dimensions.
- A static corpus cannot fully capture the dynamics, freshness, or adversarial nature of the live web.
- Quantitative results in the post's original summary were not reproduced here; consult the paper PDF before citing numbers.
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
Key points
Why reproducible retrieval matters
Classic IR evaluation (e.g., on BEIR or MS MARCO) uses fixed corpora and judged relevance, giving stable metrics like nDCG@10. Agentic deep research breaks this setup: the agent formulates its own queries against a moving target (the live web or a commercial API), so reported task success rates conflate agent capability with uncontrollable environmental drift. A frozen corpus plus an open search index restores the fixed-environment assumption while keeping the task realistic and open-ended.