MindSearch: An AI Search Framework That Mimics Human Thinking

*Si-Suo — Deep web information seeking and integration through a multi-agent framework.*

Introduction

Information seeking and integration is a complex cognitive task. While search engines have reshaped how we retrieve information, they often struggle with complex human intents. Inspired by advances in large language models (LLMs), recent work combines LLMs with search engines, but these approaches face three challenges:

• Complex requests cannot be accurately and completely retrieved by a search engine in one shot.
• The information needed is scattered across multiple web pages mixed with heavy noise.
• Large amounts of long-form web content can quickly exceed LLM context length limits.

MindSearch simulates the human thought process in web information seeking and integration via a simple yet effective LLM-based multi-agent framework consisting of WebPlanner and WebSearcher.

Overall Architecture

• WebPlanner acts as a high-level planner: it simulates human reasoning, decomposes the user query into atomic sub-questions as nodes in a graph, and progressively expands the graph based on search results from WebSearcher.
• WebSearcher handles each sub-question, performing hierarchical information retrieval and summarizing valuable information back to the planner.

This multi-agent design lets MindSearch search and integrate information from over 300 web pages in parallel within 3 minutes—equivalent to about 3 hours of human work. Built on GPT-4o or InternLM2.5-7B, it shows significant gains in response depth and breadth on both closed-set and open-set QA.

WebPlanner: Planning via Graph Construction

Simply prompting an LLM to plan an entire data workflow yields poor results—current LLMs struggle to decompose complex problems and understand their topological relations. MindSearch therefore models problem-solving as a directed acyclic graph (DAG): given a question Q, the solving trajectory is G(Q) = ⟨V, E⟩, where each node is an independent web search (plus auxiliary START and END nodes) and edges encode reasoning topology.

Leveraging LLMs' strong code generation, the model interacts with the graph by writing code using predefined atomic functions (adding nodes/edges). Each round, the LLM reads the conversation—including previously generated code and search results—then outputs thoughts plus new code executed via a Python interpreter. Added nodes trigger WebSearcher queries. This "code as planning" process improves control and data flow in long-context scenarios.

WebSearcher: Hierarchical Web Browsing

WebSearcher is a RAG agent with internet access that summarizes valuable responses from search results. To handle massive web content within limited context, it uses a coarse-to-fine selection strategy:

1. Query rewriting – the LLM generates several similar queries to broaden coverage and improve recall. 2. Search content aggregation – queries are executed via search APIs (Google, Bing, DuckDuckGo), returning URLs, titles, and snippets. 3. Detailed page selection – results are merged by URL; the LLM selects the most valuable pages for in-depth reading. 4. Final summarization – full content of selected pages is fed to the LLM, which generates the answer.

This hierarchical retrieval makes browsing hundreds of pages tractable and enables efficient extraction of highly relevant, detailed information.

Multi-Agent Context Management

Because WebPlanner delegates search tasks to independent agents and relies only on their summaries, it can focus purely on question decomposition, while each WebSearcher focuses only on its assigned sub-query. This clear role separation substantially reduces context computation and offers an efficient long-context management solution for LLMs.

> "MindSearch gathers and integrates relevant information from over 300 pages in less than 3 minutes, whereas human experts require roughly 3 hours for a similar cognitive workload."

Experimental Results

All models accessed the internet only via the Bing Search API. Evaluations covered:

• Open-set QA: 100 curated real-world human queries; five human experts compared MindSearch (InternLM2.5-7b-chat) against Perplexity.ai (Pro) and ChatGPT with search on depth, breadth, and factuality. Human evaluators preferred MindSearch responses over ChatGPT-Web (GPT-4o) and Perplexity Pro.
• Closed-set QA: benchmarks including Bamboogle, Musique, and HotpotQA, with both GPT-4o and InternLM2.5-7b-chat backends. MindSearch improved over baselines by 4.7% on GPT-4o and 6.3% on InternLM2-7b.

Applications

• Academic research: rapid cross-disciplinary information gathering and synthesis.
• Business decisions: comprehensive market analysis and competitive intelligence.
• Education: deep, comprehensive learning materials.
• Journalism: support for in-depth investigative reporting with multi-perspective synthesis.

Future work may extend MindSearch to visual inputs and interactive web browsing. More info: https://github.com/InternLM/MindSearch