RNG-Bench: Evaluating Multimodal LLMs in Controllable Non-Markovian Games

Forum topic · 小凯 · 2026-06-19

Summary

RNG-Bench (Reconstructive Non-Markov Games) is a benchmark suite designed to isolate a multimodal model's ability to reconstruct past observations and act on them during multi-step interaction. It features two complementary games: Matching Pairs, where briefly revealed card identities must be recalled later, and 3D Maze, where egocentric views must be integrated into a spatial map. Both games run in a unified harness with three controlled difficulty axes: grid size, visual pattern, and observation modality. The benchmark adds a head-to-head duel protocol to control instance-level variance and a Memory Gap metric that disentangles forgetting from poor action selection. The hardest configurations require roughly 128K-token contexts and 350 image inputs per episode and remain far from saturated by frontier multimodal LLMs. Analysis shows most residual errors stem from forgetting earlier observations rather than suboptimal decision making. Fine-tuning Qwen3.5-9B on optimal-policy rollouts and filtered demonstrations improves RNG-Bench performance and transfers to existing benchmarks without degrading general multimodal capability.

Paper Overview

Field: Computer Vision Authors: Shengyuan Ding, Xilin Wei, Xinyu Fang arXiv: 2506.14982

Background

Deploying multimodal foundation models as closed-loop policies increasingly requires conditioning actions on observations that are no longer visible. Existing benchmarks either expose the full state (conflating hidden-state reconstruction with other agent skills) or test recall only after an episode has ended.

RNG-Bench

RNG-Bench (Reconstructive Non-Markov Games) isolates a base model's ability to reconstruct past observations and act on them during multi-step interaction. It includes two complementary games:

1. Matching Pairs: card identities briefly revealed at specific locations must later be recalled. 2. 3D Maze: egocentric views must be integrated into a spatial map.

Both games are evaluated under a unified harness with three controlled difficulty axes: grid size, visual pattern, and observation modality. The benchmark also introduces:

A head-to-head duel protocol to control instance-level variance
A Memory Gap metric that disentangles forgetting from poor action selection

Original Abstract

> Deploying multimodal foundation models as closed-loop policies increasingly requires conditioning actions on observations that are no longer visible. However, existing benchmarks either expose the full state, conflate hidden-state reconstruction with other agent skills, or test recall only after an episode has ended. We introduce RNG-Bench (Reconstructive Non-Markov Games), a benchmark suite designed to isolate a base model's ability to reconstruct past observations and act on them during multi-step interaction. RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map. Both games are evaluated under a unified harness with three controlled difficulty axes: grid size, visual pattern, and observation modality. The benchmark further introduces a head-to-head duel protocol to control for instance-level variance and a Memory Gap metric that disentangles forgetting from poor action selection. The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs. Memory Gap analysis shows that most residual errors stem from forgetting earlier observations rather than from suboptimal decision making. Finally, fine-tuning Qwen3.5-9B on optimal-policy rollouts and filtered model demonstrations improves performance on RNG-Bench and transfers to existing benchmarks without degrading general multimodal capability.

Key Findings

The hardest configurations require ~128K-token contexts and 350 image inputs per episode; frontier MLLMs remain far from saturation.
Most residual errors come from forgetting earlier observations, not from suboptimal decision making.
Fine-tuning Qwen3.5-9B on optimal-policy rollouts and filtered model demonstrations improves RNG-Bench performance and transfers to existing benchmarks without degrading general multimodal capability.