Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Overview
| Field | Content | |------|------| | Title | Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge | | Authors | Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, et al. (7 authors) | | Published | 2018-03-14 | | Source | https://arxiv.org/abs/1803.05457 | | Type | Academic paper |
One-line summary
This paper introduces ARC (AI2 Reasoning Challenge), a new question answering benchmark that demands far more powerful knowledge and reasoning than prior challenges such as SQuAD or SNLI.
Background and Motivation
Large-scale QA benchmarks up to 2018 could largely be solved with shallow heuristics such as retrieval plus word co-occurrence matching. To measure genuine progress in knowledge and reasoning, the authors assembled a new dataset of natural, grade-school science questions (originally authored for human standardized tests). The goal is to encourage AI research in advanced question answering that goes beyond pattern matching.
Key Contributions
- A new question set of 7,787 natural grade-school science questions, the largest public-domain set of this kind.
- The set is partitioned into a Challenge Set (questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm) and an Easy Set.
- Baseline evaluations showing that leading neural models from SQuAD and SNLI fail to significantly outperform a random baseline on the Challenge Set.
- Release of the ARC Corpus, 14 million science sentences relevant to the task, plus implementations of the three neural baseline models.
- ARC exposed the gap between reading-comprehension benchmarks (where neural models excelled) and true reasoning-heavy QA.
- The Challenge Set became a widely used leaderboard for measuring advances in commonsense and science reasoning, later popularized further as ARC-Challenge in evaluations of large language models.
- The accompanying 14M-sentence science corpus supports research in knowledge retrieval and explanation-based QA.
- AI Search Has A Citation Problem, Mar 2025, CJR
- Evaluation of Retrieval-Augmented Generation: A Survey, May 2024
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles
- ARES: An Automated Evaluation Framework for RAG
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
- Original paper: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457
Baselines and Evaluation
The authors tested several baselines on the Challenge Set, including leading neural models transferred from the SQuAD and SNLI tasks. None could significantly outperform a random baseline, reflecting the difficult reasoning and knowledge requirements of the task. Precise numerical results should be verified against the original paper's tables.
Original Abstract (preserved)
> We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.