OpenAI Introduces SimpleQA: A Benchmark for Measuring Model Factuality and Hallucinations (Oct 2024)

Forum topic · 小凯 · 2026-07-05

Summary

This forum post summarizes OpenAI's October 2024 release of SimpleQA, a benchmark designed to measure the factuality of language models on short, fact-seeking questions. SimpleQA addresses a key challenge in LLM evaluation: models often produce incorrect answers with high confidence (hallucinations), and existing benchmarks either saturate quickly or are costly to grade. The benchmark consists of short factoid questions where the correct answer is known, enabling automated grading that checks whether the model's response matches a reference answer. It is specifically designed to evaluate three behaviors: whether the model answers correctly, whether it abstains when uncertain, and whether it hallucinates by stating wrong answers confidently. This makes it a practical tool for tracking progress in model reliability and calibration. The forum post places SimpleQA in the broader context of evaluating search and retrieval-augmented systems, where hallucination rates and citation accuracy are critical quality metrics. It also cross-references related evaluation work such as surveys on retrieval-augmented generation evaluation, the ARES automated evaluation framework, and datasets of information-seeking questions. The post includes engineering checklists covering latency, safety, and cost considerations for deploying retrieval-based systems.

OpenAI Introduces SimpleQA: A Benchmark for Measuring Model Factuality (Oct 2024)

Source

Title: Introducing SimpleQA
Organization: OpenAI
Date: October 2024
URL: https://openai.com/index/introducing-simpleqa/
Category: Evaluation of Search Engines

What is SimpleQA

SimpleQA is a benchmark released by OpenAI in October 2024 to measure the factuality of language models on short, fact-seeking questions. Its central motivation is that models frequently answer incorrect questions with high confidence — the hallucination problem — and that frontier models have saturated many older short-form QA benchmarks, making it hard to measure further progress.

SimpleQA is designed around three key properties:

1. Easy to grade — questions have short, verifiable answers, so correctness can be checked automatically against a reference answer without expensive human labeling. 2. Hard for frontier models — the questions target knowledge where models commonly err, so the benchmark does not saturate immediately. 3. Hallucination-aware — grading distinguishes between a correct answer, an incorrect answer, and a failure to answer (abstention), allowing measurement of both accuracy and calibration.

This makes SimpleQA a practical metric for tracking how often a model produces confident factual errors versus appropriately declining to answer.

Context in Search and RAG Evaluation

The post situates SimpleQA within the broader landscape of information retrieval and retrieval-augmented generation (RAG) evaluation. As systems shift from static retrieval pipelines to LLM-driven and agentic search, evaluation metrics move beyond classic ranking measures (nDCG, MRR, Recall@k) toward task-level signals such as answer correctness, citation accuracy, and hallucination rate — exactly the dimensions SimpleQA targets.

Takeaways for Practitioners

Researchers: SimpleQA offers a reproducible factuality baseline; report abstention and error rates separately, not just accuracy.
Engineers: Hallucination rate, latency, and cost are hard production constraints; combine automated factuality grading with human audits and citation checks.
Product teams: User-perceivable reliability (correct answers, honest abstentions) matters more than offline ranking metrics alone.

*Note: For exact question counts and numeric model scores, consult the original announcement at openai.com.*