Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks

Forum topic · 小凯 · 2026-09-13

Summary

A new survey paper by Mia Lassiter and Brinnae Bent (arXiv:2609.11018) addresses the lack of a standard definition for the term 'agent' in artificial intelligence, a gap that complicates evaluation, comparison, and reproducibility in AI agent research. The survey is organized around five dimensions of agenticness: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. For each dimension, the authors examine how the underlying capability has been conceptualized in prior work and synthesize the metrics, benchmarks, and evaluation frameworks used to assess it. The review offers a structured account of the current agent evaluation landscape, highlighting established approaches as well as areas where evaluation remains limited or inconsistent. The authors also introduce the Agent Compendium, a public-facing digital resource that organizes and extends the evaluation methods identified in the survey. Together, the survey and compendium provide a common structure for evaluating and comparing agent capabilities across AI systems, supporting more reproducible research, clearer communication, and more systematic study of artificial agents. Fields: cs.AI, cs.MA.

Paper Overview

Research areas: cs.AI, cs.MA Authors: Mia Lassiter, Brinnae Bent Published: 2026-09-13 arXiv: 2609.11018

Abstract

The term *agent* in artificial intelligence lacks a standard definition, complicating the evaluation, comparison, and reproducibility of AI agent research. The authors address this ambiguity through a survey organized around five dimensions of agenticness:

1. Environmental interaction 2. Learning and adaptation 3. Autonomy 4. Goal-directed behavior 5. Temporal coherence

For each dimension, the paper examines how the underlying capability has been conceptualized across prior work and synthesizes the metrics, benchmarks, and evaluation frameworks used to assess it. The review provides a structured account of the current landscape of agent evaluation, highlighting both established approaches and areas where evaluation remains limited or inconsistent.

Key Contributions

Five-dimension framework for characterizing agenticness in AI systems
Structured synthesis of existing metrics, benchmarks, and evaluation frameworks per dimension
Identification of gaps where agent evaluation remains limited or inconsistent
Agent Compendium: a public-facing digital resource that organizes and extends the evaluation methods identified through the review

Together, the survey and compendium provide a common structure for evaluating and comparing agent capabilities across AI systems, supporting more reproducible research, clearer communication, and more systematic study of artificial agents.