PIQA: Reasoning About Physical Commonsense in Natural Language (arXiv 1911.11641)

Overview

This post is an index entry from a curated awesome-list on search, recommendation, and LLM evaluation. It references the paper "PIQA: Reasoning about Physical Commonsense in Natural Language".

| Field | Detail | |-------|--------| | Title | PIQA: Reasoning about Physical Commonsense in Natural Language | | Authors / Affiliations | Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi | | Published | November 2019 (arXiv) | | Link | https://arxiv.org/abs/1911.11641 | | Resource type | Academic paper | | List section | Evaluation of Search engines |

What PIQA Is

PIQA (Physical Interaction Question Answering) is a benchmark for physical commonsense reasoning in natural language. The task: given an everyday goal (e.g., making a sandwich, cleaning a tool), a model must select the more appropriate of two candidate solutions. Solving it requires grounding knowledge of how objects are physically used, manipulated, and organized — precisely the knowledge that text-only training corpora often underrepresent.

The paper's motivation is building AI systems that can follow procedural instructions for real-world tasks (cooking, crafting, repairs), described in the original abstract as a step toward machines that "can build things" from natural-language plans.

Key Contributions

• A binary-choice QA benchmark targeting physical commonsense, designed so that superficial language cues do not trivially reveal the correct answer.
• A data pipeline harvesting goal–solution pairs from instructables.com, followed by human curation of validation/test splits, including deliberately similar solution pairs to increase difficulty.
• Baseline evaluations showing that contemporary language models (e.g., GPT-era pretrained models) score substantially below human performance, quantifying the gap in physical commonsense.
• The benchmark has since become a standard LLM evaluation suite component (commonly grouped with ARC, HellaSwag, WinoGrande as physical/social/commonsense probes).

Context in the Curated List

The entry sits in the "Evaluation of Search engines" section, cross-referenced with related entries:

• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv 2105.03011)
• ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv 2401.13131)
• AI Search Has A Citation Problem (CJR, March 2025)

Editorial Notes

This forum post is largely a template-generated summary; several of its sections (method taxonomy, experiment protocols, engineering checklist) describe generic IR/RAG practice rather than the PIQA paper's specific content. The post itself instructs readers to consult the original PDF for exact quantitative results before citing numbers. For authoritative details — including the reported human baseline, model scores, and full dataset statistics — see arXiv:1911.11641.