Paper Overview
- Research Area: Computer Vision (CV)
- Authors: Mingxian Lin, Shengju Qian, Yuqi Liu
- Published: 2025-06-06
- arXiv: 2506.04867
- OmniGameArena: a real-time benchmark of twelve newly built Unreal Engine 5 games spanning Solo (7), PvP (3), and Coop (2) modes, with unified action interfaces across all games.
- Improvement Dynamics Curve (IDC): an agentic-reflection harness in which a tool-using reflector LLM autonomously refines a bounded skill prompt across multiple rounds.
- arXiv: https://arxiv.org/abs/2506.04867
Abstract
Vision-language model (VLM) agents are increasingly deployed in interactive game environments. Yet game benchmarks for VLM agents typically report a single first-attempt score per (agent, game) pair, focus on single-agent Solo play, and lack unified protocols for evaluating heterogeneous agent classes (commercial VLMs, open-weight VLMs, and specialized game policies) on the same footing.
The authors address these gaps with:
Beyond cold-start leaderboard scores, IDC exposes two additional observable metrics for each (agent, game) pair:
1. How scores evolve across reflection rounds. 2. How learned skills perform on held-out task variants.
The paper reports results for 12 VLM agents on the cold-start leaderboard, as well as results for 4 top-performing agents under IDC.