Odysseus: Scaling VLMs to 100+ Turn Game Decision-Making via Reinforcement Learning

Forum topic · 小凯 · 2026-05-04

Summary

Odysseus is a reinforcement learning framework that trains vision-language models (VLMs) to perform long-horizon decision-making over 100+ turns in visually rich game environments, notably Super Mario Land. Most existing approaches either rely on large-scale human demonstrations via supervised fine-tuning or restrict RL training to short 20–30 turn horizons, which yields only local, reactive policies. Odysseus combines VLM-based visual understanding with end-to-end RL, requiring no human demonstration data, so the model autonomously explores, plans paths, and executes multi-step strategies across entire game levels. The forum post explains why long-horizon RL is hard—credit assignment over hundreds of steps, massive action spaces prone to local optima, and the need for stable visual state representations across frames—and argues that Odysseus's milestone is demonstrating that VLMs can learn sustained planning and execution purely through RL rather than imitation. The author frames this as a shift from reflexive, short-sighted agents toward true planning intelligence, closer to human gameplay, with broader implications for interactive and general-purpose AI. Paper: arXiv 2605.00347.

Overview

This forum post introduces Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning.

Paper: Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning
Authors: Chengshuai Shi, Wenzhe Li, Xinran Liang, Yizhou Lu, Wenjia Yang, Ruirong Feng, Seth Karten, Ziran Yang, Zihan Ding, Gabriel Sarch, Danqi Chen, Karthik Narasimhan, Chi Jin
arXiv: 2605.00347 | 2026-04-29

The Problem: AI Games Only Last a Few Dozen Turns

A human playing Super Mario Bros. observes the screen, understands the scene, plans a path, executes actions, and sustains this loop for 100+ turns to complete a level. Current VLM methods fall short:

They either need large amounts of human demonstrations (SFT), or
RL training is limited to 20–30 turn horizons.

Short-horizon RL learns only local policies—like a nearsighted player who reacts to what's directly ahead without a long-term plan—so complex tasks remain out of reach. Combining visual understanding with long-term planning is difficult.

What Odysseus Does

Core idea: use reinforcement learning to train vision-language models for 100+ turn long-horizon decision-making in visual environments.

Technical approach:

1. VLM + RL — the VLM understands game frames while RL optimizes long-term strategy, trained end-to-end. 2. Long-horizon decisions — 100+ turn interactions with long-term, goal-directed planning rather than short-sighted local optimization, coordinating perception and action. 3. Super Mario Land — a visually rich environment requiring scene understanding, path planning, and action execution in a complete decision loop. 4. No human demonstrations — pure RL training without SFT dependence; the agent learns autonomously through exploration and exploitation.

The contrast: short-horizon RL learns "jump when you see an enemy," while Odysseus learns the structure of the level—where to jump, where to run, what to stomp—global planning with sustained execution.

Why Long-Horizon Decision-Making Is Harder

Credit assignment: after 100 turns, which step earned the reward? Attribution is difficult.
Exploration: the action space is enormous, long-term consequences are uncertain, and local optima are common.
Visual understanding: every frame must be interpreted; states change rapidly, requiring stable visual representations.

Why it's worth it:

Completing whole levels requires global planning, not just local reactions—genuinely intelligent behavior.
Humans are long-horizon decision-makers; Odysseus moves toward human-level play.
Learned policies generalize better—they reflect understanding rather than memorization.

Feynman-Style Reflection

The post invokes Feynman: knowing the name of something is different from truly understanding it. Applied to AI decision-making:

> An AI that reacts for 20 steps is "reflex"; an AI that plans for 100 steps is "thinking." Odysseus's milestone is proving that VLMs can learn long-horizon decision-making via pure RL—not memorizing human demonstrations, but understanding, planning, and executing on their own. That is a step toward general intelligence.

Takeaways

If you research VLMs or RL, ask:

1. Does my model only make short-horizon decisions? 2. Can pure RL train long-horizon behavior? 3. Does visual understanding support long-term planning? 4. How is the credit assignment problem solved?

Odysseus reminds us: real AI is not a "reaction machine" but a "planning agent." When a VLM sustains decisions over 100+ turns, it evolves from a "visual recognizer" into a "game player"—understanding scenes, planning paths, executing strategy. In interactive AI, the best model is not the most accurate one, but the one that can keep acting the longest. In the long river of decision-making, short-term splashes matter less than the long-term heading.