SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines

Forum topic · 小凯 · 2026-07-05

Summary

SuperGPQA (arXiv:2502.14739) is a large-scale benchmark designed to evaluate the graduate-level knowledge and reasoning abilities of large language models across 285 disciplines spanning 13 fields of study. Built by the M-A-P team with 97 contributors, the benchmark targets the long tail of academic knowledge—covering specialized and less commonly studied fields where LLM performance is hardest to assess—rather than only popular domains like mathematics or coding. The dataset contains tens of thousands of graduate-level multiple-choice questions sourced from publicly available materials such as textbooks and exams. The authors evaluated dozens of frontier LLMs, including proprietary and open-weight models, and reported substantial performance gaps, particularly in niche disciplines, showing that even the strongest models leave significant room for improvement. To mitigate hallucination and improve reliability, the framework incorporates search-augmented inference so models can consult external knowledge before answering. SuperGPQA is released openly to support reproducible evaluation and to serve as a reference point for measuring progress on broad, deep scientific knowledge in LLMs. This forum post indexes the paper within an information-retrieval and LLM-evaluation collection, alongside related work on RAG evaluation and agent benchmarks.

SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines

Paper: SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (arXiv, Feb/Mar 2025)

Team: M-A-P Team, with Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, and 97 contributors in total.

Overview

SuperGPQA is a graduate-level knowledge and reasoning benchmark for large language models (LLMs). Its distinguishing feature is breadth: it spans 285 graduate disciplines across 13 fields of study, deliberately covering the *long tail* of academia—specialized, low-resource subjects that popular benchmarks (which cluster around math, code, and general reasoning) largely ignore.

Key points

Scale and coverage: Tens of thousands of graduate-level multiple-choice questions distributed across 285 disciplines, constructed from publicly available sources such as textbooks and exams.
Long-tail focus: Emphasis on disciplines where existing benchmarks provide little signal, exposing gaps in LLM knowledge that headline scores hide.
Broad model evaluation: The authors evaluate a wide range of frontier LLMs (proprietary and open-weight) and find that even top models show substantial room for improvement, with performance dropping notably in niche disciplines and a clear gap between large and smaller models.
Search-augmented inference: To reduce hallucination and leverage external knowledge, the framework includes search-based inference support, connecting the benchmark to retrieval-augmented (RAG) evaluation practices.
Open release: The dataset and evaluation pipeline are released openly to enable reproducible benchmarking of scientific knowledge in LLMs.

Why it matters for the IR/RAG community

Evaluating LLMs on specialized graduate knowledge is closely tied to retrieval: for long-tail domains, parametric knowledge alone is unreliable, and search-augmented or retrieval-augmented answering becomes essential. SuperGPQA therefore fits alongside work on RAG evaluation and agentic search benchmarks, providing a knowledge-depth axis that complements task-success metrics.

Related entries in this collection

Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (arXiv:2105.03011)
ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13131)

> Original abstract and full results: https://arxiv.org/abs/2502.14739

*Note: quantitative score tables should be verified against the paper PDF; this post summarizes the paper's scope and findings based on the abstract and public metadata.*

SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines

Summary

SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines

Overview

Key points

Why it matters for the IR/RAG community

Related entries in this collection

Tags