Key points
- Benchmark purpose. GameCraft-Bench (arXiv:2606.17861) is the first benchmark requiring a coding agent to build a complete, playable game end-to-end inside a real game engine (Godot 4) and validate playability through actual interaction replay.
- Three desiderata. Existing benchmarks fail on at least one of: Engine Grounding (runs in a real engine), Artifact Completeness (deliverable is a full launchable project), and Interactive Verification (validated by replayed gameplay, not static checks).
- Task suite. 140 tasks across 15 game families (Platformer, Strategy, Tycoon, Open-world, Roguelike, Visual Novel, Puzzle, Shooter, Simulation, Card Game, Horror, Rhythm, Idle, Racing, Sports), each authored and reviewed by 12 experienced annotators with hidden rubrics and oracle solutions.
- Five-stage pipeline. Task Packaging → Agent Generation → Build Gate → Replay (1280×720 viewport, sampled at 2 fps) → Multimodal Scoring by a GPT-5.5 judge across four weighted dimensions: Core Mechanics 15%, Content Depth 35%, Functional Visuals 15%, Art & Presentation 35%. Final score = BUILD × weighted sum.
- Headline result. Best agent (Claude Code / Opus-4.7 high) achieves only 41.46% overall; Codex / GPT-5.5 high 39.49%; Kimi Code / Kimi-K2.6 30.65%; Code Buddy / MiMo-V2.5-Pro 24.10%; Code Buddy / GLM-5.1 18.29%; Code Buddy / MiniMax-M2.7 10.95%; Codex / DeepSeek-V4-Pro 2.15%. Core Mechanics scores are uniformly higher than Content Depth, Visuals, and Art—prototypes are easier than finished games.
- Visual feedback matters. Kimi-K2.6 used screenshots 2,998 times across 140 tasks (~21.4/task) versus 268 for GPT-5.5, illustrating perception–action iteration as a key debugging loop.
- Tool-call volume ≠ quality. MiMo-V2.5-Pro averages 128 tool calls/task (56.3% shell), yet its tool-call count correlates almost zero with task score (r = +0.016); zero-score tasks typically stem from failing to submit demo traces.
- Dimension decoupling. For Kimi-K2.6, Mechanics↔Depth correlates at r=0.61 and Mechanics↔Visuals at r=0.53, but Art↔Visuals only r=0.11—functional visuals and artistic presentation are weakly coupled skills.
- Judge reliability. Repeated GPT-5.5 judging shows standard deviations around 0.0037–0.0050 per family; the judge is slightly more lenient than humans on Content Depth and Art but stricter on Functional Visuals.
- Comparison vs. related work. Unlike GameDevBench (partial edits), OpenGame-Bench (web games, static/page-level checks), and WebGameBench (web games, browser interaction), GameCraft-Bench uniquely combines Godot engine grounding, full project completeness, and replay-based interactive verification.
- Limitations and future work. Scope is 2D + Godot only; audio is not directly evaluated; the multimodal judge may carry model/API bias; "fun" is not measured. Future directions include 3D/Unity/Unreal coverage, audio-aware judging, long-horizon progression, human–agent collaboration, and cross-engine comparisons.
- "Writing code" is not "making a game"; the dominant bottleneck is content depth and visual presentation, not mechanics.
- Perception–action loops (screenshot-based debugging) should be a first-class capability in future game-generation agents.
- Hidden rubrics plus full-project deliverables are necessary to prevent reward hacking in creative-system benchmarks.
- Paper: https://arxiv.org/abs/2606.17861
- Project site: https://tongxuluo.github.io/gamecraft-bench-website