OpenAI Introduces SimpleQA: A Benchmark for Factuality and Hallucination Measurement

Source: https://openai.com/index/introducing-simpleqa/

Overview

In October 2024, OpenAI released SimpleQA, a benchmark for measuring the factuality of language models on short, factoid-style questions. The central motivation is that models frequently produce fluent but incorrect answers, and existing evaluations do not adequately capture both *what models get right* and *whether models know what they don't know*.

Key points

• Task format: The benchmark consists of short factoid questions with a single verifiable ground-truth answer, making grading straightforward and automatable with an LLM-based grader.
• Dataset scale: SimpleQA contains over 4,000 question–answer pairs, written and verified by human annotators.
• Adversarial curation: Questions were selected adversarially against earlier model generations, so items that models already answered correctly were filtered out — keeping the benchmark challenging over time.
• Calibration matters: Beyond raw accuracy, SimpleQA evaluates calibrated confidence. Models are asked to state how confident they are in each answer, enabling metrics such as accuracy under selective prediction (F1 at optimal abstention threshold).
• Key finding: Even frontier models answer a minority of SimpleQA questions correctly while showing significant overconfidence — they often assign high confidence to wrong answers.

Evaluation methodology

• Responses are graded automatically against reference answers, judging correctness rather than style.
• Confidence scores allow computation of calibration curves and selective prediction metrics, rewarding models that abstain when uncertain.
• The short-answer format avoids the ambiguity and grading instability common in long-form factuality benchmarks.

Relevance to search and retrieval evaluation

SimpleQA fits into a broader ecosystem of retrieval and RAG evaluation work, including:

• Evaluation of Retrieval-Augmented Generation: A Survey
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Article Abstracts
• ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation
• AI Search Has A Citation Problem (CJR, Mar 2025)

For search-augmented systems, SimpleQA offers a clean yardstick for hallucination rates with and without retrieval, though its short-answer scope does not cover multi-hop reasoning, citation quality, or long-form synthesis.

Caveats

• The benchmark emphasizes single-answer factoids; real-world search quality involves nuance SimpleQA does not capture.
• Adversarial filtering against public models means scores may shift as knowledge coverage evolves.
• Quantitative results should be verified against the original announcement and accompanying paper.