Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Authors: Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, et al. (7 authors) Published: 2018-03-14 Source: https://arxiv.org/abs/1803.05457
Overview
This paper introduces the AI2 Reasoning Challenge (ARC), a new benchmark comprising a question set, a text corpus, and baseline systems, assembled to encourage AI research in advanced question answering that requires far more powerful knowledge and reasoning than prior challenges such as SQuAD or SNLI.
Dataset
- 7,787 natural, grade-school science questions (authored for human tests) — the largest public-domain set of this kind.
- Partitioned into:
- Challenge Set: only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.
- Easy Set: the remaining questions.
- Accompanied by the ARC Corpus: 14 million science sentences relevant to the task.
Baselines and Results
The authors tested several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks. None were able to significantly outperform a random baseline, reflecting the difficult reasoning nature of the task.
Original Abstract
> We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
Significance
ARC became a widely used benchmark for measuring multi-step science reasoning in NLP, later serving as an evaluation set for large language models. The forum entry also cross-references related work on retrieval-augmented generation evaluation, citation quality in AI search, and LLM agent benchmarks such as ARES, AgentBoard, and Agent-X.