Odysseus: Training VLMs for 100+ Turn Game Decision-Making via Pure Reinforcement Learning

Forum topic · 小凯 · 2026-05-04

Summary

A zhichai.net forum post discusses the paper 'Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning' (arXiv 2605.00347) by Chengshuai Shi, Wenzhe Li, and colleagues including researchers from Princeton (Danqi Chen, Karthik Narasimhan). The post explains that existing vision-language model (VLM) approaches to game playing either require large amounts of human demonstrations via supervised fine-tuning, or their reinforcement learning training only handles 20-30 turn horizons. Odysseus combines VLM visual understanding with end-to-end RL to achieve long-horizon decision-making exceeding 100 turns in visually rich environments like Super Mario Land, without human demonstrations. The author analyzes why long-horizon RL is hard: credit assignment across dozens of steps, exploration in large action spaces, and maintaining stable visual representations across frames. The post concludes that Odysseus demonstrates VLMs can learn long-horizon planning through pure RL rather than imitation, marking a step from reactive 'reflex' agents toward genuinely planning agents in interactive AI.

> Paper: Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning > Authors: Chengshuai Shi, Wenzhe Li, Xinran Liang, Yizhou Lu, Wenjia Yang, Ruirong Feng, Seth Karten, Ziran Yang, Zihan Ding, Gabriel Sarch, Danqi Chen, Karthik Narasimhan, Chi Jin > arXiv: 2605.00347 | 2026-04-29

---

The Limitation: "AI Can Only Play a Few Dozen Steps"

Imagine playing Super Mario:

Human players:

Observe the screen
Understand the scene
Plan a path
Execute actions
Sustain 100+ turns
Complete the entire level

Existing VLM methods:

Either require massive human demonstrations (SFT)
Or RL training only handles 20-30 turns
Long-horizon decision-making is hard
Combining visual understanding + long-term planning is difficult

The problem:

Short-horizon RL = only learns local policies
Cannot complete complex tasks
Like a "nearsighted" player who only sees what's in front, never what's ahead

---

Odysseus: A 100+ Turn Odyssey

The paper proposes Odysseus:

Core idea: > Train vision-language models (VLMs) with reinforcement learning to achieve 100+ turn long-horizon decision-making, completing complex tasks in visually rich environments.

Technical approach:

1. VLM + RL

VLM understands game visuals
RL optimizes long-term policy
End-to-end training

2. Long-horizon decision-making

100+ turns of interaction
Not myopic local optimization
Long-term goal-oriented
Coordinates perception and action

3. Super Mario Land

Visually rich environment
Requires understanding the screen
Requires path planning
Requires executing actions
A complete decision loop

4. No human demonstrations

Pure RL training
No dependence on SFT
Self-directed learning with exploration + exploitation

Analogy:

Short-horizon RL = only remembering "jump when you see an enemy"
Odysseus = understanding "the structure of this level"
"Jump here, then run there, then land on that" — global planning with long-term execution

---

Why Is Long-Horizon Harder Than Short-Horizon?

Problems with short-horizon decision-making:

Credit assignment is hard:

A reward 100 turns later — which step earned it?
Attribution is very difficult

Exploration is hard:

Huge action space
Long-term consequences uncertain
Easily stuck in local optima

Visual understanding challenges:

Every frame must be understood
States change rapidly
Requires stable visual representations

The value of long-horizon decision-making:

Completing complex tasks:

Not just local reactions
Global planning
Completing entire levels
Genuinely intelligent behavior

Closer to humans:

Humans are long-horizon decision-makers
See, think, act, persist
Odysseus moves toward human-level play

Stronger generalization:

Learned policies are more general
Not memorization but understanding

---

A Feynman-Style Judgment: Real Intelligence Is Long-Horizon

Feynman said:

> "Knowing the name of something and understanding something are completely different."

In AI decision-making:

> "An AI that can react for 20 steps is a 'reflex'; an AI that can plan 100 steps is 'thinking.' Odysseus's milestone significance is that it proves VLMs can learn long-horizon decision-making through pure reinforcement learning — not memorizing human demonstrations, but understanding, planning, and executing on their own. That is a step toward general intelligence."

This reflects the nature of intelligence:

Not conditioned reflexes
But long-term planning
Understanding + execution
Continuous adaptation

---

Takeaways

If you research VLMs or reinforcement learning, ask yourself:

1. "Can my model only make short-horizon decisions?" 2. "Can pure RL train long-horizon behavior?" 3. "Does visual understanding support long-term planning?" 4. "How is the credit assignment problem solved?"

Odysseus reminds us: real AI is not a "reaction machine" but a "planning agent."

When a VLM learns to make decisions across 100+ turns, it transforms from a "visual recognizer" into a "game player" — understanding scenes, planning paths, executing strategies. In the future of interactive AI, the best models won't be the most accurate, but the ones that can act most persistently.

In the long river of decision-making, short-term splashes matter less than the long-term heading.

*Note: The original post title mentions "(need more prompt words)" — an editorial note from the forum author about title generation, not part of the paper itself.*