From Reaction to Intelligence: Why Your VLM Agent Should Drop the SFT Playbook

Forum topic · 小凯 · 2026-05-06

Summary

A zhichai.net forum post discusses the Odysseus paper (arXiv:2605.00347), which argues that reinforcement learning, not supervised fine-tuning (SFT), is the key to scaling vision-language models (VLMs) to long-horizon decision-making. Using Super Mario Land as a testbed requiring 100+ turns of sequential visual decisions, the authors (Shi, Li, Liang, Lu, Yang, Feng, Karten, Yang, Ding, Sarch, Chen, Narasimhan, and Jin) show that an RL-trained VLM achieves roughly 3x the average game progress of an SFT model trained on human gameplay recordings. The core challenge is credit assignment over 100+ turns: critic-free methods like GRPO and Reinforce++ become unstable at this horizon because group-relative reward signals drown in variance. Odysseus pairs PPO with a lightweight turn-level critic that predicts expected return each turn, enabling precise penalty allocation. Pretrained VLMs (e.g., Qwen2.5-VL) supply an action prior that drastically improves sample efficiency, so RL refines rather than discovers basic game mechanics. The resulting agents generalize within a game (across levels) and across games while retaining general-domain capabilities. The post concludes that industry-wide SFT-first pipelines may be a costly mistake: SFT is a starting point, but RL is the engine for superhuman long-horizon agents.

Overview

In 2025, two teams each taught an AI to play Super Mario Land:

Team A (SFT): Fine-tuned a VLM on 1,000 hours of human player recordings. The model learned to mimic humans — jump at enemies, grab coins. It looked smart, but human skill was its ceiling.
Team B (RL): Used the same VLM without SFT, letting it play directly with reward for completing levels and penalties for dying. A lightweight critic told it each turn how far it was from finishing.

Three months later, Team B's model achieved 3x the average game progress of Team A.

This illustrates the finding of the paper *Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning* (arXiv:2605.00347), submitted May 1, 2026: for teaching AI to play games, SFT is the coach, RL is the engine.

Why SFT Fails at Long Horizons

Games like Super Mario Land require 100+ turns of continuous decision-making — every step must account for what happened before and what may come next. Jumping at an enemy is a reaction; knowing when to risk a hidden path is intelligence. SFT can only teach reaction, because human player performance is the ceiling.

The core difficulty is credit assignment: if the agent dies on turn 100, was it the decision at turn 5, turn 50, or turn 99?

SFT: No answer — the human demonstrator didn't die there.
GRPO: Computes a single signal after 100 turns — too sparse.
PPO + turn-level critic: A "prophet" provides a value estimate every turn.

The Turn-Level Critic

The paper's solution is a lightweight turn-level critic that tells the agent, given the current screen, how many points it expects to ultimately earn. If the critic predicts 1,000 points but the agent only earns 100, the penalty can be precisely assigned to the specific turns where things went wrong, rather than delivering one blanket "you died" after 100 turns.

In RL terms, the critic is a value function estimating expected cumulative return:

V(s_t) = \mathbb{E}\left[\sum_{k=t}^{T} \gamma^{k-t} r_k \Big| s_t\right]

PPO vs. GRPO at Long Horizons

PPO (Proximal Policy Optimization) uses a clipped surrogate objective with advantages $$A(s,a) = Q(s,a) - V(s)$$ computed by a critic. GRPO (Group Relative Policy Optimization, used by DeepSeek-R1) is critic-free, estimating advantages from relative rewards across multiple sampled outputs. GRPO works on short tasks (20–30 turns), but at 100+ turns the within-group variance becomes too large and the signal drowns in noise. The turn-level critic provides a stable baseline, letting PPO converge reliably in long-horizon settings.

The Pretrained VLM as an Action Prior

RL alone isn't enough — from scratch, an agent needs millions of attempts to discover even basic mechanics. A pretrained VLM (e.g., Qwen2.5-VL) already knows:

Which sprite is Mario
To jump when enemies approach from the left
That coins are positive and pits are fatal

It provides not a final policy but an action prior — a distribution over reasonable actions. RL optimizes on top of this prior:

\pi_{\text{RL}}(a|s) \propto p_{\text{VLM}}(a|s) \cdot \exp(Q(s,a))

It's like having someone who can already drive learn racing — no need to relearn the steering wheel, only how to take corners fastest.

Results

At least 3x average game progress over SFT baselines
Consistent improvement in in-game generalization (different levels of the same game) and cross-game generalization (different games)
General-domain capabilities preserved — the model doesn't become a game-only specialist

Implications for Industry

The most unsettling point: the entire VLM agent industry may be doing the wrong thing — spending millions annotating human trajectory data for SFT. The cost of hiring players, recording trajectories, and cleaning data can be ten times the cost of RL training, and results can never exceed human performance.

Odysseus doesn't say SFT is useless. It says: SFT is the starting point, RL is the destination. Pretrained VLMs supply common sense and visual understanding; SFT can bootstrap quickly; but superhuman long-horizon decision-making requires RL. Companies invested in SFT data annotation may want to redirect budgets toward RL infrastructure and critic design.

Paper Details

| Item | Content | |:-----|:--------| | Title | Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning | | Authors | Chengshuai Shi, Wenzhe Li, Xinran Liang, Yizhou Lu, Wenjia Yang, Ruirong Feng, Seth Karten, Ziran Yang, Zihan Ding, Gabriel Sarch, Danqi Chen, Karthik Narasimhan, Chi Jin | | arXiv ID | 2605.00347 | | Published | May 1, 2026 | | Category | cs.LG (Machine Learning) | | Core method | PPO + lightweight turn-level critic, pretrained VLM as action prior | | Environment | Super Mario Land (100+ turn visual decision-making) | | Key findings | ≥3x average game progress; in-game and cross-game generalization; preserved general-domain ability | | Baselines | GRPO, Reinforce++ (critic-free, unstable at long horizons) |

Concept index

| Concept | Description | |:--------|:------------| | Credit Assignment | Determining which action in a long-horizon task is responsible for the final outcome | | Turn-level Critic | Lightweight per-turn value predictor enabling precise credit assignment | | PPO | Proximal Policy Optimization; uses a clipped objective and a critic | | GRPO | Group Relative Policy Optimization; critic-free, uses within-group relative rewards | | Action Prior | Distribution over reasonable actions supplied by the pretrained VLM | | Advantage function | $$A(s,a) = Q(s,a) - V(s)$$ , measuring an action relative to average |