Meituan LongCat Launches LoHoSearch Benchmark, Dragging Search Agents From 90% Down to 34.74%
Source: Meituan LongCat / arXiv 2606.12837 / FreeAI Help URL: https://x.com/Meituan_LongCat/status/2078119654632124547 Date: 2026-07-17
What Happened
On July 17, Meituan's LongCat team published the LoHoSearch paper and benchmark (arXiv:2606.12837). Using a Wikipedia knowledge graph with 7.62 million entities, it automatically generates questions that expose the true capability level of search agents:
- Best score among 11 frontier models: 34.74% — the second tier sits at 15-16%
- The same models already score 90%+ on BrowseComp — a 55-point gap
- Context-management strategies gain only +6.8pp on LoHoSearch, versus +14pp on BrowseComp
- Solving LoHoSearch questions requires on average 74% more tool calls than BrowseComp
- BrowseComp questions are hittable in a few retrieval steps, so context management mainly helps by "reducing redundant searches and accelerating convergence."
- LoHoSearch requires multi-hop, multi-structure reasoning; its marginal benefit is cancelled by the problem's inherent search space and structural complexity.
- Mainstream agent frameworks systematically underestimate the true cost of long-horizon tasks in their tool-call budget design.
- The bottleneck for long-horizon search agents is not "whether they can search" but "whether they can sustain long-horizon reasoning plus context management."
- Dual judges still carry bias — both models come from "BrowseComp-style" training lineages and may score conservatively on structured/multi-hop problems. Independent third-party validation is needed.
- 544 questions is a small sample — statistical significance for 1-2pp differences between frontier models may be insufficient.
- KG scale ceiling — 7.62M entities is far smaller than full Wikidata; coverage of vertical domains (medicine, law) may be thin.
- Training-data contamination — decontamination details were not provided; if models saw samples in pretraining, the evaluation loses meaning.
- Meituan LongCat on X: https://x.com/Meituan_LongCat/status/2078119654632124547
- arXiv paper: https://arxiv-vanity.com/papers/2606.12837
- Hugging Face dataset: https://huggingface.co/datasets/meituan-longcat/LoHoSearch
- In-depth commentary: https://freeai.help/blog/browsecomp-yi-bei-shua-bao-mei-tuan-longcat_zh
The dataset is open-sourced on Hugging Face at meituan-longcat/LoHoSearch: 544 questions / 11 domains / tree + graph structures / 7.62M Wikipedia entities + 265 million edges.
Why BrowseComp Collapsed So Fast
BrowseComp was released by OpenAI in April 2025 as a search-agent benchmark: hard questions with easily verifiable answers. Deep Research debuted at 51.5%, human testers scored 33.3% — the de facto standard at the time.
Ten months later, top models climbed from 30% to 90%+. Anthropic's Claude Mythos Preview System Card even noted that "this benchmark is rapidly losing discriminative power."
The LongCat team's explanation is blunt: BrowseComp was written by humans. Human annotation has two unavoidable limitations:
1. Annotators unconsciously pick familiar entities — familiarity means high popularity, direct connections, and answers reachable in a few retrieval steps. 2. Annotators lack a global view — they cannot see "how many candidate entities this relation actually has across the whole knowledge graph."
The result: the difficulty ceiling is capped by the cognitive horizon of human annotators.
LoHoSearch's Solution: Hand Question Construction to the Knowledge Graph
The automated construction pipeline:
| Step | Action | |---|---| | 1. Build KG | Large-scale knowledge graph from Wikipedia: 7.62M entities + 265M edges | | 2. Global edge selection | Use global entity statistics to pick edges with large search spaces | | 3. Subgraph sampling | Tree structures expand search space; graph structures introduce cyclic dependencies | | 4. Answer uniqueness check | Answers are unique at the KG level | | 5. Natural language formulation | LLM converts subgraphs into natural-language questions | | 6. Multi-round validation | Automated + human review for correctness and uniqueness |
Key design: search-space size + structural complexity — the two core dimensions of search-problem difficulty. Humans cannot see the global distribution of these during annotation; machines can.
544 questions is not many, but each is a "structured retrieval maze" — you must follow trees and navigate graph cycles, not just hit a keyword.
Results: Frontier Models Fall Back to One-Third
| Model | LoHoSearch accuracy | BrowseComp accuracy (same period) | |---|---|---| | GPT-5.5 (best) | 34.74% | ~90% | | Ranks 2-4 | 15-16% range | ~90% | | Remaining 7 models | Below 15% | — |
Note: best 34.74%, second tier drops straight to 15-16%. Not "collectively slightly worse" — a cliff. This means LoHoSearch's difficulty is not a uniform gradient but a clear capability threshold.
Why Context Strategies "Fail"
On BrowseComp, context-management strategies (summarizing, discarding, resetting) yield an average +14pp. On LoHoSearch, the same strategies yield only +6.8pp — halved gains.
Why?
Implication: prompt and context engineering cannot close capability gaps on a benchmark like LoHoSearch.
74% More Tool Calls: The Real Cost of Long-Horizon Reasoning
Another finding: correct solutions require 74% more tool calls than BrowseComp. This means:
Dual-Judge Scoring: GPT-4.1 + Qwen2.5-32B Averaged
To avoid single-LLM-as-judge bias, LoHoSearch averages scores from GPT-4.1 and Qwen2.5-32B, measuring true performance and confidence-calibration error at extreme difficulty — a methodological upgrade for 2026 agent benchmarks.
Why It Matters
1. Shifts search-agent evaluation from "human intuition-driven" to "data-metric-driven" — a generational switch in benchmark paradigm, aligned with SWE-bench → SWE-bench Pro, CEO-Bench, and CivBench. 2. The cliff exposes a structural blind spot — 34.74% vs 90% is a warning signal for every company building Deep Research products. 3. Context strategies underdeliver — the 6.8pp vs 14pp comparison means prompt engineering yields half the benefit at this difficulty. 4. Open source, downloadable data — the full 544-question dataset is on Hugging Face; any team can run it. A strong candidate for a 2026 "must-run" search-agent benchmark. 5. Meituan LongCat's presence in agent evaluation — Chinese teams are now helping lead the design of high-difficulty agent benchmarks.
Risks and Open Questions
One-Line Takeaway
BrowseComp went from 30% to 90% in ten months; LongCat's 7.62M-entity knowledge graph dragged frontier search agents back to 34.74%, with the second tier at 15-16% and context-strategy gains halved to +6.8pp. This is not a harder exam — it touches a systemic blind spot of current search agents.
Source links: