Can Language Models Play the Wikipedia Game? Testing LLM Information Retrieval via Link Navigation

Forum topic · 小凯 · 2026-10-13

Summary

A 2026 arXiv paper (2610.13948) by Alexander Hickey, Joanna Knight, and James Bishop tests language models' information retrieval abilities using the Wikipedia Game, where players must reach a target article using only article links. The authors compare sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets. They build very large local graphs of Wikipedia pages and paper citations using OpenAlex, providing code to construct and navigate them so research on model navigation does not depend on third-party APIs. Models are benchmarked against human and TF-IDF baselines, and the Wikipedia Game format enables robust comparisons across models, languages, and domains. Results show no model significantly outperforms humans, but the best models (GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B) match human success rates. Performance and rankings vary by language and topic, offering practical guidance for practitioners deploying retrieval with language models.

Paper Overview

Field: Machine Learning Authors: Alexander Hickey, Joanna Knight, James Bishop Published: 2026-10-09 arXiv: 2610.13948

Abstract

Language models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. The authors used the Wikipedia Game to test language models on an analogous information retrieval task: in the game, players must reach a target article using only article links.

Key Contributions

Compared sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets.
Constructed very large local graphs of Wikipedia pages and paper citations using OpenAlex, with provided code to build and navigate them — enabling the study of language models' navigation capabilities without relying on third-party APIs.
Benchmarked models against human and TF-IDF baselines.
The Wikipedia Game format supports robust comparisons across models, languages, and domains.

Findings

No model significantly outperformed humans, but the best-performing models — GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B — achieved success rates comparable to humans.
Performance and rankings vary by language and topic, which is relevant for practitioners choosing models for real-world retrieval tasks.

Original Abstract (excerpt)

> Language models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. We used the Wikipedia Game to test language models on an analogous information retrieval task. In the game, players must reach a target article using only article links. We compared sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets. We construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex and provide code to build and navigate them. This allows the study of language models' navigation capabilities without relying on third-party APIs. We compared the models with human and TF-IDF benchmarks...

Read the full paper: arXiv:2610.13948