Summary
OmniGameArena is a real-time benchmark of twelve newly built Unreal Engine 5 games designed to evaluate vision-language model (VLM) agents under a unified protocol. It covers Solo (7 games), PvP (3 games), and Coop (2 games) modes with unified action interfaces, addressing limitations of existing VLM game benchmarks that report only a single first-attempt score per (agent, game) pair and lack comparable evaluation across heterogeneous agent classes such as commercial VLMs, open-weight VLMs, and specialized game policies. The authors also introduce the Improvement Dynamics Curve (IDC), an agentic-reflection harness in which a tool-using reflector LLM autonomously refines a bounded skill prompt across multiple rounds. IDC exposes two additional observable metrics beyond cold-start leaderboard scores: how scores evolve across reflection rounds and how learned skills transfer to held-out task variants. The paper reports cold-start leaderboard results for 12 VLM agents and IDC results for 4 top-performing agents. Authored by Mingxian Lin, Shengju Qian, and Yuqi Liu, it is available on arXiv as 2506.04867.
Paper Overview
Field: Computer Vision (CV)
Authors: Mingxian Lin, Shengju Qian, Yuqi Liu
Published: 2025-06-06
arXiv: 2506.04867
Abstract
Vision-language model (VLM) agents are increasingly deployed in interactive game environments. Yet game benchmarks for VLM agents typically report a single first-attempt score per (agent, game) pair, focus on single-agent Solo play, and lack unified protocols for evaluating heterogeneous agent classes (commercial VLMs, open-weight VLMs, and specialized game policies) on the same footing.
The authors address these gaps with OmniGameArena, a real-time benchmark of twelve newly built Unreal Engine 5 games spanning:
- Solo: 7 games
- PvP: 3 games
- Coop: 2 games
All games feature unified action interfaces, enabling fair comparison across agent classes.
Alongside the benchmark, they introduce the Improvement Dynamics Curve (IDC), an agentic-reflection harness in which a tool-using reflector LLM autonomously refines a bounded skill prompt across multiple rounds. Beyond cold-start leaderboard scores, IDC exposes two additional observable metrics for each (agent, game) pair:
1. How scores evolve across reflection rounds.
2. How learned skills perform on held-out task variants.
The paper reports cold-start leaderboard results for 12 VLM agents, as well as IDC results for 4 top-performing agents.
---
*Automatically collected on 2026-06-10.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177981039