Don't Mask the Environment: Observation Supervision (ActObs) Changes How Agents Learn in RL

Paper Overview

Field: NLP Authors: Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah Published: 2026-09-17 arXiv: 2609.20715

Abstract (translated from the post)

Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. The authors ask whether this convention provides the best initialization for subsequent reinforcement learning. They introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences — without adding data, parameters, sequence tokens, or forward passes.

The methods perform similarly after SFT but diverge after GRPO:

• On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget on Terminal-Bench 2.0.
• On Qwen3-8B, it trades a small amount of pass@1 reliability for higher pass@k (pass@16 +3.4 pp) and solves more distinct tasks.
• The advantage extends to the cross-domain code-editing benchmark aider-polyglot (+4.2 pp pass@1 at 4B) — tasks unseen in both SFT and RL phases.

Analysis shows ActObs retains more entropy during RL and requires smaller policy updates, keeping the final policy closer to the SFT initialization. The gap traces back to SFT: action and observation gradients rapidly orthogonalize, and pure-action training leaves large residual observation gradients, degrading environment-prediction ability below the base model. Joint supervision prevents this one-sided specialization, preserving consequence-prediction capability and preparing the policy for downstream exploration.

--- *Auto-collected on 2026-09-20*