GAIA: A Benchmark for General AI Assistants (arXiv, Nov 2023)

Overview

• Paper: GAIA: a benchmark for General AI Assistants
• Authors / Affiliations: Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom (Meta AI, Hugging Face, AutoGPT)
• Link: https://arxiv.org/abs/2311.12983
• Category: Evaluation of Search engines / LLM agent evaluation

Key Points

• GAIA poses questions that are conceptually simple for humans but hard for most advanced LLMs: answering typically requires a few logically ordered steps, akin to what a competent human could manage with basic tools.
• The benchmark tests a combination of capabilities: multi-step reasoning, web browsing, code execution, file and image processing, and factual accuracy — mirroring how a general AI assistant would operate in real life.
• Questions are grounded in real, verifiable facts and have short, unambiguous answers, allowing objective, automated scoring rather than subjective judge models.
• The dataset contains 466 questions across three difficulty levels, built through a multi-stage annotation process with validation to ensure answerability and uniqueness.

Headline Results

• Humans achieve ~92% accuracy on GAIA.
• GPT-4 with plugins reaches only ~15%, and even strong task-specific agents fall notably short of human performance.
• This contrasts with traditional benchmarks like MMLU, where LLMs already exceed average human performance — suggesting existing evaluations measure narrow academic knowledge rather than real-world assistant competence.

Why It Matters

• GAIA argues that progress toward useful general assistants should be measured by robustness, tool use, and reasoning in realistic settings, not only on knowledge-heavy static tests.
• The benchmark highlights the gap between language-model fluency and agentic reliability: planning, retrieving up-to-date information, and executing actions correctly.
• GAIA has become a widely used reference evaluation for agentic AI systems and is hosted on Hugging Face.

Relevance to Search / IR

• Within LLM-era information retrieval, GAIA exemplifies the shift from static metrics (nDCG, MRR) toward task success rates, tool-call correctness, and citation/answer verifiability for agentic search systems.
• It connects with related evaluation work such as ARES, AgentBoard, and RAG evaluation surveys.

Reference

• arXiv: https://arxiv.org/abs/2311.12983