From History to State: Constant-Context Skill Learning Lets LLM Agents Skip Rereading Instructions

Forum topic · 小凯 · 2026-05-08

Summary

This post explains the paper 'From History to State: Constant-Context Skill Learning for LLM Agents' (arXiv:2605.05413) from Arizona State University researchers, using a Feynman-style analogy: a great chef doesn't reread the whole cookbook every meal. The method moves reusable procedural knowledge from prompts into model weights via Context-to-Weights: a frozen base model plus a lightweight LoRA skill module (~0.5-0.7GB, ~2% parameters updated) per task family, paired with a deterministic rule-based task tracker that maintains a compact state block. At inference, the agent conditions only on the current observation, one-step history, and the state block, keeping context constant regardless of episode length. Trained with step-level SFT followed by subgoal-guided GRPO-style RL (rewards derived from tracker state, no LLM judge), the Qwen3-8B agent reaches 89.6% success on ALFWorld (unseen), 76.8% on WebShop, and 66.4% on SciWorld, while reducing per-turn prompt tokens 2-7x versus ReAct and total episode tokens roughly 10-14x.

From History to State: Why the Best Chefs Don't Reread the Whole Cookbook

A Feynman-style deep dive into the paper "From History to State: Constant-Context Skill Learning for LLM Agents" by Haoyang Xie, Xinyuan Wang, Yancheng Wang, Puda Zhao, and Feng Ju (Arizona State University), arXiv:2605.05413, released 2026-05-06.

The Problem: Privacy-Cost-Capability Trilemma

LLM agents are becoming real assistants (browsing, files, coding, tools), but:

Cloud models are capable but leak sensitive intermediate state to external APIs
Local models preserve privacy but are weaker and fail on long-horizon tasks
Both reprocess long skill prompts and ever-growing interaction histories every step

Under ReAct, the context is roughly |c_t^hist| ≈ |g| + |d_k| + Σ|o_i| + |a_i| — prompt length grows linearly with the episode.

Core Idea: Context-to-Weights

Move reusable procedural knowledge from context into weights:

1. Freeze the base model θ₀ 2. Learn a lightweight skill module (LoRA adapter, φ_k) per task family 3. At inference, condition only on: current observation + one-step history + a compact state block, with |x_t| ≤ B_k bounded regardless of trajectory length

Like a chef who has internalized the cookbook into muscle memory, the agent only checks today's ingredients (observation), the last step (one-step context), and the kitchen's current state (state block: completed steps, acquired items, remaining tasks).

The Deterministic Tracker

The state block is not LLM-summarized. A rule-based deterministic tracker maintains structured task progress:

ALFWorld: target object, holding status, destination receptacle, checked locations
WebShop: current query, inspected product, selected/remaining options, purchase readiness

The tracker is reproducible and requires no extra model calls. Crucially, the same tracker state both renders the state block and provides subgoal rewards during RL, keeping optimization aligned with what the model sees.

Training Pipeline

Stage 1 — Step-Level SFT:

Collect expert trajectories; convert them into step-level (x_t, a_t*) pairs via the tracker
x_t = Format(g, o_t, q_t, b_t): instruction, current observation, one-step history, state block
Train a LoRA adapter with standard next-action supervision (W_k = W_0 + (α/r)·B_k·A_k, ~2% of parameters, 0.5–0.7GB per skill module)

Stage 2 — Subgoal-Guided RL:

Sample K rollouts; the tracker provides environment reward (r_env), subgoal progress reward (r_prog), and error penalties for invalid/repeat/reverting actions (r_err)
GRPO-style group-normalized policy gradient updates, with the SFT adapter frozen as a reference
Reward rules were designed offline with GPT-5.5, then implemented as deterministic rules — no LLM judge needed

Results (Qwen3-8B, SFT+RL)

| Benchmark | Seen | Unseen | |---|---|---| | ALFWorld | 83.6% | 89.6% | | WebShop | – | 76.8% | | SciWorld | 72.8% | 66.4% |

Context efficiency (per-turn prompt tokens):

| Method | ALFWorld | WebShop | SciWorld | |---|---|---|---| | ReAct full | 1,310 | 3,093 | 1,938 | | Ours | 184 | 488 | 496 |

A 2–7x reduction in prompt tokens, and roughly 10–14x fewer total episode tokens on ALFWorld (34K → 3K) and WebShop (47K → 3.4K).

Ablations (WebShop): current observation alone yields 1.2% success; + one-step context: 5.6%; + state block without training: 23.6%; after SFT: 62.2%; SFT+RL: 76.8%. The state block plus training into weights is what makes it work.

Takeaway

The method's elegance is in learning what to *forget*: put "how to do things" into weights, and keep only "where am I now" in context. As the authors put it: "A recurring workflow should not require the agent to reread the same procedure, instructions, and examples at every execution."

References

1. Xie, H., et al. (2026). From History to State: Constant-Context Skill Learning for LLM Agents. *arXiv:2605.05413*. 2. Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. *ICLR 2023*. 3. Shridhar, M., et al. (2021). ALFWorld: Aligning Text and Embodied Environments. *NeurIPS 2021*. 4. Yao, S., et al. (2022). WebShop: Scalable Real-World Web Interaction. *EMNLP 2022*. 5. Hu, E. J., et al. (2022). LoRA: Low-Rank Adaptation. *ICLR 2022*. 6. Shao, Z., et al. (2024). DeepSeekMath. *ICLR 2024*.