GameHorizon Suite: Multi-Horizon Data and Evaluation Benchmark for AI Gameplay

Forum topic · 小凯 · 2026-09-23

Summary

GameHorizon Suite is a unified data and evaluation framework designed to measure AI gameplay capabilities across multiple temporal horizons. The suite consists of three components: GameHorizon-Annotator, a scalable automated pipeline for multi-horizon instruction annotation; GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions spanning 5,000 hours of recordings across 21 games collected by 100 expert human players; and GameHorizon-Bench, which provides reproducible offline and step-wise online testing. The offline track uses thousands of standardized questions covering three core tasks and diagnostic variants, while the online track verifies whether offline scores reflect actual gameplay ability and localizes failures to specific steps in long-horizon gameplay. Using the suite, the authors evaluated 47 models through over one million model calls, revealing a hierarchy of task difficulty and notable capability differences between model families. The dataset, annotator, and benchmark will be open-sourced.

Overview

Research area: Computer Vision (CV) Authors: Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, Cheng Cheng, Wangbo Yu, Honghao Chen, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan Release date: 2026-09-21 arXiv: 2609.25001

Abstract

Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, and precise action control over multiple temporal horizons. Existing datasets and benchmarks, however, either cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts.

To address these challenges, the authors introduce GameHorizon, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families.

Components

1. GameHorizon-Annotator — A scalable, automated annotation pipeline for multi-horizon instructions. 2. GameHorizon-Data — The first large-scale AAA gameplay dataset, built with the annotator pipeline. It contains temporally aligned video, player actions, and multi-horizon instructions, covering 5,000 hours of recordings across 21 games, collected by 100 expert human players. 3. GameHorizon-Bench — Reproducible offline and step-wise online testing:

Offline track: evaluation via thousands of standardized questions spanning three core tasks and a set of diagnostic variants.
Online track: checks whether offline scores reflect actual gameplay ability and localizes failures to specific steps in long-horizon gameplay.

Evaluation Results

Using GameHorizon Suite, the authors evaluated 47 models through more than one million model calls, revealing:

A clear hierarchy of task difficulty across temporal horizons.
Significant differences in capabilities between model families.

The work aims to provide a standardized yardstick for assessing gameplay abilities across temporal horizons and model families. The dataset, annotator, and benchmark will be open-sourced to support future research.

---

*Auto-collected on 2026-09-23*