CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (ACL 2019)
Overview
This post summarizes the ACL 2019 paper CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge by Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant (Tel-Aviv University).
- Paper page: <https://aclanthology.org/N19-1421/>
- Venue: NAACL 2019 (ACL Anthology N19-1421)
- Task type: multiple-choice question answering (5 options)
- Dataset: 12,102 multiple-choice questions, each with five answer options.
- Construction: Crowd workers author questions based on ConceptNet relations, using a target concept and its neighboring nodes to create questions that probe commonsense associations.
- Validation by difficulty: Before collecting answers, the authors test a strong baseline — BERT-large pretrained on RACE — and require it to score below a set threshold, filtering out questions solvable via textual cues alone.
- Questions about familiar people and objects are easier for models; abstract relations remain hard.
- Noise in ConceptNet affects question quality and serves as an upper-bound concern for the dataset.
- The paper proposes a knowledge-augmented two-encoder baseline, and subsequent work has built extensively on CommonsenseQA as a standard evaluation set.
- Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019). CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. NAACL 2019. <https://aclanthology.org/N19-1421/>
Motivation
Modern QA systems can often answer questions by retrieving evidence from text, but many questions require prior knowledge — the kind of everyday commonsense humans acquire without explicit instruction. The authors design a benchmark where the answer cannot be inferred from a passage, forcing models to rely on learned world knowledge.
Key points
Results
| Model | Accuracy | |-------|----------| | Random | ~20% | | BERT-large (RACE-pretrained) | ~56% | | Human | ~89% |
The large gap between the neural baseline and humans shows the benchmark genuinely requires commonsense reasoning, not just reading comprehension.
Analysis
Why it matters
CommonsenseQA helped shift the NLP community's attention from extractive QA toward reasoning-centric evaluation, complementing datasets like SWAG, PIQA, and later BIG-bench tasks. It remains widely cited in LLM evaluation suites.