Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks

Summary

A new survey by Mia Lassiter and Brinnae Bent (arXiv:2609.11018) addresses the lack of a standard definition for the term "agent" in artificial intelligence, which complicates evaluation, comparison, and reproducibility in AI agent research. The authors organize the survey around five dimensions of agenticness: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. For each dimension, they examine how the capability has been conceptualized in prior work and synthesize the metrics, benchmarks, and evaluation frameworks used to assess it. The review maps the current landscape of agent evaluation, highlighting both established approaches and areas where evaluation remains limited or inconsistent. The authors also introduce the Agent Compendium, a public digital resource that organizes and extends the evaluation methods identified through the review. Together, the survey and compendium offer a common structure for comparing agent capabilities across AI systems, supporting more reproducible research and clearer communication.