SFR-DeepResearch: Reinforcement Learning for Autonomously Reasoning Single Agents

Introduction

Imagine an explorer with only a crude map, a compass, a knife, and a notebook, venturing alone through unknown jungle. A similar scenario is unfolding in AI: single-agent large language models (LLMs) are being equipped to reason and use tools autonomously while exploring complex problems. This article summarizes the paper *SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents* (arXiv:2509.06283v2, September 9, 2025), which proposes a novel approach that improves single-agent Deep Research (DR) capability through continuous reinforcement learning (RL).

Unlike traditional multi-agent systems, a single agent requires no predefined roles or static workflows — it dynamically decides its next move from context. The research trains a single agent with a minimal toolset (web search, web browsing, and a Python interpreter), targeting 28.7% performance on the Humanity's Last Exam benchmark.

Background and Related Work

Evolution of deep research agents. DR agents answer complex factual questions, typically relying on internet access and tool use such as code execution. Since early systems like Search-R1, reasoning LLMs (e.g., the o3 system) have integrated web browsing and code interpretation for tasks ranging from short Q&A to abstract report writing. Existing systems fall into two categories:

• Single-agent (e.g., OpenAI's DeepResearch): simple and flexible, one LLM autonomously invokes tools without intermediate instructions, suitable for unknown tasks.
• Multi-agent (e.g., OpenManus): divides labor (planner, coder, etc.) for efficiency, but is constrained by predefined workflows.

The authors' conviction is that single agents are more promising due to their generality, and can be seamlessly integrated into multi-agent systems while avoiding redundant complexity.

The SFR-DeepResearch Method

SFR-DR combines a lean agent reasoning scaffold with a custom RL training recipe.

1. Agent reasoning scaffold

Three basic tools:

• search_internet(query): returns the top 10 search results (URL, title, description).
• browse_page(url, section_id): fetches page content and converts it to Markdown, removing hyperlinks to force the agent to discover new URLs through search.
• code_interpreter(code): executes stateless Python locally with a 5-minute timeout, no file system or sensitive package access.

Memory management: the agent reserves part of the fixed context length as a memory buffer; when the limit is exceeded, it invokes a cleanup tool to select key information — ensuring a flexible effective context window.

2. RL training recipe

• Synthetic data: a dataset of complex search and reasoning tasks that challenge existing DR agents.
• Algorithmic improvements: built on REINFORCE, adding temporal advantage normalization and trajectory filtering to stabilize multi-turn optimization.

Experiments were run on QwQ-32B, Qwen3-8B, and gpt-oss-20B models.

Results and Analysis

SFR-DR-20B achieves 28.7% on Humanity's Last Exam, outperforming same-scale baselines. Tool-call trajectories show the agent iterating through search, browsing, and coding to reach an answer. On QwQ-32B and Qwen3 models, multi-turn dialogues were reconstructed as single-turn context tasks while preserving "thinking" tokens to improve reasoning efficiency.

Ablations show memory management and length normalization prevent tool-call degradation. SFR-DR is also competitive on FRAMES and GAIA, demonstrating robustness. Notably, memory management — dynamically pruning non-critical information — resembles an optimization of human short-term memory and is especially important for long-context tasks.

Discussion and Future Directions

SFR-DR demonstrates that a single agent can substantially improve autonomous reasoning via continuous RL, particularly under limited resources. However, the deliberately minimal toolset demands strong intrinsic reasoning; overly complex tools may weaken training effectiveness.

Future plans:

• Introduce confidence scoring to optimize tool selection.
• Extend to multimodal tasks combining images and text.

References (selected)

1. Agarwal, M. and Khanna, D. When persuasion overrides truth in multi-agent LLM debates: Introducing a confidence-weighted persuasion override rate (cw-por), 2025. https://arxiv.org/abs/2504.00374 2. Amayuelas, A., et al. Multiagent collaboration attack. arXiv:2406.14711, 2024. 3. Brown, T., et al. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020. 5. Cobbe, K., et al. Training verifiers to solve math word problems, 2021. https://arxiv.org/abs/2110.14168 6. Du, Y., et al. Improving factuality and reasoning in language models through multiagent debate, 2023. https://arxiv.org/abs/2305.14325 8. Hendrycks, D., et al. Measuring massive multitask language understanding, 2021. https://arxiv.org/abs/2009.03300 9. Irving, G., et al. AI safety via debate, 2018. https://arxiv.org/abs/1805.00899 14. Sharma, M., et al. Towards understanding sycophancy in language models, 2023. https://arxiv.org/abs/2310.13548 17. Wang, X., et al. Self-consistency improves chain of thought reasoning in language models, 2022. https://arxiv.org/abs/2203.11171

Conclusion

Like a lone explorer conquering complex problems with wisdom and simple tools, SFR-DeepResearch shows that a single agent, trained with an innovative RL framework, can reach 28.7% benchmark performance — challenging the complexity of multi-agent systems and opening a new chapter in single-agent deep research.