HellaSwag: Can a Machine Really Finish Your Sentence? (2019) — Paper, Code & Dataset

Forum topic · 小凯 · 2026-07-05

Summary

HellaSwag is a 2019 benchmark introduced by Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi (Allen Institute for AI / University of Washington) to test commonsense language inference through sentence completion. Available on arXiv (1905.07830), the benchmark asks whether machines can finish a sentence in a plausible, commonsense-consistent way. HellaSwag was constructed via adversarial filtering: wrong endings are generated by language models yet rated by humans as trivially easy to identify while fooling state-of-the-art models. The dataset contains over 37,000 validation and test examples sourced from ActivityNet captions and WikiHow articles. The paper's central finding was that while humans achieve near 95% accuracy, the best models of 2019 scored well below humans (BERT around 47%), demonstrating a significant gap in commonsense reasoning. The benchmark became a standard evaluation suite item for large language models, and modern LLMs have since closed much of this gap. The post aggregates links to the arXiv paper, source code, and dataset for the community.

HellaSwag: Can a Machine Really Finish Your Sentence? (2019)

Paper: https://arxiv.org/abs/1905.07830

Authors: Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi

Resources: arXiv paper + code + dataset

Overview

HellaSwag is a benchmark for studying commonsense language inference: the task of finishing a sentence or short paragraph so that the ending is plausible and consistent with everyday common sense. The title asks directly whether a machine can really finish your sentence — and in 2019, the answer was largely no.

Key ideas

Adversarial dataset construction: Wrong (implausible) endings are machine-generated using language models, then filtered adversarially so they fool state-of-the-art NLP models but are trivially easy for humans to reject.
Scale and sources: HellaSwag contains over 37,000 validation and test examples, drawn from ActivityNet video captions (physical, grounded activities) and WikiHow articles (procedural, abstract knowledge).
Human–machine gap: Humans achieve roughly 95% accuracy, while the best 2019 models (including BERT-large) scored far lower (around 47%), showing that pretrained models lacked robust commonsense inference rather than just surface-level pattern matching.

Why it matters

HellaSwag became a standard component of large language model evaluation suites (e.g., alongside Winogrande and ARC). It demonstrated that benchmarks can be built to resist memorization and statistical shortcuts via adversarial filtering. Later LLMs substantially narrowed the human–machine gap on this benchmark, making it a useful historical marker for tracking commonsense reasoning progress.

Cross-references

Evaluation of Retrieval-Augmented Generation: A Survey (May 2024, arXiv:2405.07437)
ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)