Don't Mask the Environment: Observation Supervision Changes How Agents Learn from RL

Forum topic · 小凯 · 2026-09-20

Summary

This paper introduces ActObs, a supervised fine-tuning (SFT) modification for LLM agents that applies loss to environment observation tokens in agent trajectories, in addition to the standard loss on agent-authored action tokens. Although deployed agents never generate observations, learning to predict them encourages the policy to model the consequences of its actions, with no extra data, parameters, sequence tokens, or forward passes. After SFT the two approaches perform similarly, but they diverge under GRPO reinforcement learning: on Qwen3-4B, GRPO initialized from ActObs achieves higher pass@k at every sampling budget on Terminal-Bench 2.0; on Qwen3-8B, it trades slight pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. Gains extend to the out-of-domain code-editing benchmark aider-polyglot (+4.2 pp pass@1 at 4B). Analysis shows ActObs retains more entropy during RL and requires smaller policy movement, staying closer to the SFT initialization. Action and observation gradients rapidly orthogonalize under ActObs, whereas action-only SFT leaves residual observation gradients, degrading consequence prediction below the base model; joint supervision preserves this capability and primes downstream exploration. arXiv: 2609.20715.

Paper Overview

Field: NLP (LLM agents, reinforcement learning)
Authors: Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah
arXiv: 2609.20715

Key Idea

Agent trajectories record what an agent does and what happens next. Standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, treating environment observations as context rather than prediction targets. The authors ask whether this convention gives the best initialization for subsequent reinforcement learning.

They propose ActObs, which additionally supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences — without adding data, parameters, sequence tokens, or forward passes.

Key Findings

The methods perform similarly after SFT, but diverge after GRPO:
Qwen3-4B: GRPO from ActObs achieves higher pass@k at every evaluated sampling budget on Terminal-Bench 2.0.
Qwen3-8B: small pass@1 reliability trade-off in exchange for higher pass@k (+3.4 pp at pass@16) and more distinct tasks solved.
Benefits extend to the cross-domain code-editing benchmark aider-polyglot (+4.2 pp pass@1 at 4B), whose tasks are unseen in both SFT and RL stages.
Analysis shows ActObs retains more entropy during RL and requires smaller policy movement, keeping the final policy closer to the SFT initialization.
Gradient analysis traces the difference to the SFT stage: with joint supervision, action and observation gradients rapidly orthogonalize. Action-only SFT leaves substantial residual observation gradients, degrading consequence-prediction ability below the base model; ActObs prevents this one-sided specialization and preserves consequence prediction, preparing the policy for downstream exploration.

Original Abstract (excerpt)

> Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. [...] The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its a...

---

*Auto-collected on 2026-09-20.*