CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (ACL 2019)
This entry indexes the NAACL-HLT 2019 paper CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge, available at https://aclanthology.org/N19-1421/.
Overview
CommonsenseQA is a multiple-choice question answering benchmark built specifically to evaluate commonsense knowledge — the background world knowledge humans use instinctively but that is rarely stated in text corpora. Key characteristics of the benchmark:
- Questions are constructed from ConceptNet, a large-scale commonsense knowledge graph, with crowdworkers authoring natural-language questions grounded in its relations.
- Each question offers five answer choices, with a single correct option.
- The dataset contains roughly 12,102 questions (about 9,741 train / 1,219 validation / 1,142 test).
- Baseline experiments demonstrate that models relying on surface matching or standard reading-comprehension techniques perform substantially below human accuracy, showing that the benchmark genuinely probes commonsense reasoning.
- Pre-trained language models (BERT-era and LLM-era) as a commonsense reasoning probe;
- Knowledge-graph-augmented QA systems that fuse structured knowledge (e.g., ConceptNet) with neural readers;
- Studies on question diversity, annotation quality, and human-model gaps in QA evaluation.
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv:2105.03011)
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- AI Search Has A Citation Problem (CJR, Mar 2025)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
Why It Matters
The benchmark has become a standard evaluation target for:
Placement in This Collection
The forum lists this paper under its Evaluation of Search Engines section, alongside resources on retrieval-augmented generation (RAG) evaluation, AI search citation quality, and agentic benchmarks (e.g., AgentBoard, ARES). Within the retrieval/ranking/generation pipeline framing used by the collection, CommonsenseQA serves as a dataset for measuring whether a system's language understanding extends beyond lexical or factual matching into everyday conceptual reasoning.
Note for Readers
Much of the surrounding post body consists of a generic catalog template (background, architecture, engineering checklists, glossary) rather than paper-specific analysis. For precise experimental results, dataset splits, and author information, refer to the original paper PDF at the ACL Anthology link above.