HellaSwag: Can a Machine Really Finish Your Sentence? (2019) — Paper, Code & Dataset
Overview
This forum post indexes the 2019 paper "HellaSwag: Can a Machine Really Finish Your Sentence?", a widely cited benchmark for commonsense language understanding.
| Field | Content | |---|---| | Title | HellaSwag: Can a Machine Really Finish Your Sentence? | | Authors | Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi | | Year | 2019 | | Paper | https://arxiv.org/abs/1905.07830 | | Type | Academic paper + code + dataset |
What HellaSwag Is
HellaSwag (Harder Endings, Longer contexts, Low-shot Activities for Situations with Adversarial Generations) is a sentence-completion benchmark. Given a short description of an everyday activity, a model must choose the most plausible continuation among four endings.
Key properties:
- Adversarially filtered: wrong endings are generated by language models and filtered so that they fool the models themselves, making the benchmark robust against dataset artifacts.
- Human-easy, model-hard: humans achieve near-ceiling accuracy (95%+), while the best models at the time of publication performed substantially worse, exposing the gap in machine commonsense reasoning.
- Zero-/few-shot relevance: the paper framed HellaSwag as a benchmark that could track model generalization over time as pretraining methods improve.
- Paper (arXiv): https://arxiv.org/abs/1905.07830
- Code and dataset: linked from the arXiv abstract page (AI2 landing page includes dataset downloads and leaderboard).
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv 2401.13178)
- AI Search Has A Citation Problem (CJR, March 2025)
Why It Matters
HellaSwag became a standard evaluation item in later LLM leaderboards (e.g., the original GPT-3 paper reported HellaSwag scores). Its core lesson — benchmarks must be built to resist memorization and artifact exploitation — influenced the design of subsequent commonsense and reasoning evaluations.