Meituan LongCat Launches LoHoSearch: Knowledge-Graph Benchmark Drops Frontier Search Agents to 34.74%
Source: Meituan LongCat / arXiv 2606.12837 / FreeAI Help X post: https://x.com/Meituan_LongCat/status/2078119654632124547 Date: 2026-07-17
What Happened
On July 17, Meituan's LongCat team published the LoHoSearch paper and benchmark (arXiv:2606.12837), which uses a Wikipedia knowledge graph with 7.62 million entities to automatically generate questions that measure the true capability of search agents:
- Best score among 11 frontier models: 34.74% — the second tier sits at 15-16%
- The same models score 90%+ on BrowseComp — a 55-point gap
- Context-management strategies yield only +6.8pp on LoHoSearch, versus +14pp on BrowseComp
- Solving LoHoSearch questions requires on average 74% more tool calls than BrowseComp
- BrowseComp questions are answerable in a few retrieval steps; context management's marginal benefit concentrates on "reducing redundant searches + faster convergence."
- LoHoSearch questions require multi-hop, multi-structure reasoning; the marginal benefit of context management is offset by the question's inherent search space and structural complexity.
- Mainstream agent frameworks' "tool-call budget design" systematically underestimates the real cost of long-horizon tasks.
- The bottleneck for long-horizon search agents is not knowing how to search, but sustaining long-horizon reasoning and context management.
- Dual-judge bias remains — both GPT-4.1 and Qwen2.5-32B belong to BrowseComp-style training ecosystems and may score conservatively on structured/multi-hop problems. Independent third-party validation is needed.
- 544 questions is a small sample — statistical significance for 1-2pp differences between frontier models may be insufficient.
- KG-scale ceiling — 7.62M entities is far smaller than full Wikidata; coverage in vertical domains (medicine, law) may be lacking.
- Training-data contamination — if models saw LoHoSearch samples during pretraining, the evaluation loses meaning. The team open-sourced the data, but decontamination details were not provided.
- Meituan LongCat post: https://x.com/Meituan_LongCat/status/2078119654632124547
- arXiv paper: https://arxiv-vanity.com/papers/2606.12837
- Hugging Face dataset: https://huggingface.co/datasets/meituan-longcat/LoHoSearch
- In-depth commentary: https://freeai.help/blog/browsecomp-yi-bei-shua-bao-mei-tuan-longcat_zh
The dataset is open-sourced on Hugging Face at meituan-longcat/LoHoSearch: 544 questions / 11 domains / tree + graph structures / built from a Wikipedia KG with 7.62M entities and 265M edges.
Why BrowseComp Collapsed So Fast
BrowseComp, released by OpenAI in April 2025, featured hard questions with easily verifiable answers. Deep Research debuted at 51.5% and human testers averaged 33.3% — it was the de facto standard at the time.
Ten months later, top models climbed from 30% to 90%+. Anthropic even noted in Claude Mythos Preview's System Card that the benchmark "is rapidly losing discriminative power."
LongCat's explanation is direct: BrowseComp questions are human-written.
Human annotation has two unavoidable limitations:
1. Annotators gravitate toward familiar entities — familiarity implies high popularity, direct connections, and answers reachable in a few retrieval steps. 2. Annotators lack a global view — they cannot see how many candidate entities a given relationship has across the entire knowledge graph.
Result: the difficulty ceiling is capped by human annotators' cognitive horizons.
LoHoSearch's Approach: Hand Question Construction to the Knowledge Graph
LongCat's automated construction pipeline:
| Step | Action | |---|---| | 1. Build KG | Large-scale KG from Wikipedia: 7.62M entities + 265M edges | | 2. Global edge selection | Use global entity statistics to select edges with large search spaces | | 3. Subgraph sampling | Tree structures expand search space; graph structures introduce circular dependencies | | 4. Answer uniqueness check | Subgraph answers are unique at the KG level | | 5. Natural-language formulation | An LLM converts subgraphs into natural-language questions | | 6. Multi-round validation | Automated + human review for correctness and uniqueness |
Key design: search-space size + structural complexity — the two core dimensions of search difficulty. Humans can't see their global distribution during annotation, but machines can.
544 questions is not a lot, but each one is a "structured retrieval maze" — you must follow trees and traverse graphs, not just hit a keyword.
Results: Frontier Models Fall Back to One-Third
| Model | LoHoSearch Accuracy | BrowseComp Accuracy (same period) | |---|---|---| | GPT-5.5 (best) | 34.74% | ~90% | | Ranks 2-4 | 15-16% | ~90% | | Remaining 7 models | Below 15% | — |
Note: the best is 34.74% while the second tier drops straight to 15-16%. This is not "collectively slightly worse" — it's a cliff, meaning LoHoSearch's difficulty has a clear capability threshold rather than a uniform gradient.
Why Context Strategies "Stop Working"
On BrowseComp, context-management strategies (summarize / discard / reset) yield an average +14pp gain. On LoHoSearch, the same strategies deliver only +6.8pp — benefits are cut in half.
Why?
Implication: prompting and context engineering barely close the capability gap on benchmarks like LoHoSearch.
74% More Tool Calls: The Real Cost of Long-Horizon Reasoning
Another finding: correct solution paths require 74% more tool calls than BrowseComp. This means:
Dual-Judge Scoring: GPT-4.1 + Qwen2.5-32B Averaged
To avoid single LLM-as-judge bias, LoHoSearch averages scores from two judges — GPT-4.1 and Qwen2.5-32B — measuring both real performance and confidence-calibration error at extreme difficulty. This is a methodological upgrade for 2026 agent benchmarks; single-judge bias was already criticized with BrowseComp, and LongCat implemented dual judges directly.
Why It Matters
1. Shifts search-agent evaluation from human-experience-driven to data-metric-driven — a generational switch in search benchmark design, aligned with SWE-bench → SWE-bench Pro, CEO-Bench, and CivBench. 2. A cliff-like gap exposes a systemic blind spot — 34.74% vs 90% is not "a bit harder"; it touches a structural blind spot of frontier models, a direct warning for all deep-research product teams. 3. Context strategies lose effectiveness — the 6.8pp vs 14pp comparison shows prompt engineering's returns halve at this difficulty; don't expect prompt engineering to close model capability gaps. 4. Open source, downloadable — the full 544-question dataset is on Hugging Face; any team can run this benchmark, making it a candidate "must-test" for 2026 search agents. 5. Meituan LongCat's presence in agent evaluation — following CEO-Bench (Princeton), CivBench, and Anthropic's Claude Mythos, a Chinese team (Meituan) is now helping lead the design of high-difficulty agent benchmarks.
Risks and Open Questions
One-Line Takeaway
BrowseComp went from 30% to 90% in ten months; LongCat used a 7.62M-entity knowledge graph to auto-generate questions that drop frontier search agents back to 34.74%. The second tier falls to 15-16%, and context-strategy gains are halved from +14pp to +6.8pp — this is not a harder exam, it's a systemic blind spot of current search agents. The next must-run benchmark is likely this one.
Links: