JitRL: Gradient-Free Continual Reinforcement Learning for Frozen LLM Agents

Key points

• The frozen-agent problem. Most deployed LLM agents cannot learn from mistakes because their weights are locked at inference time, leaving them unable to adapt to dynamic environments such as WebArena web tasks or Jericho text adventures.
• Two prior paths, two dead ends. Gradient-based RL frameworks like WebRL (PPO/GRPO) continually fine-tune the policy but cost more than $3,500 per Llama-3.1-70B run and still produce a static model prone to catastrophic forgetting. Training-free methods such as Reflexion, ExpeL, and Agent Workflow Memory (AWM) keep weights intact but rely on prompt-based heuristics with no formal optimization guarantee.
• Three faces of memory. Reflexion (NeurIPS 2023, arXiv:2303.11366) stored free-form verbal reflections and reached 91% pass@1 on HumanEval. ExpeL (AAAI 2024, arXiv:2308.10144) distilled reusable insights from past trajectories. AWM (arXiv:2409.07429) abstracted successful trajectories into reusable workflows and boosted WebArena relative success by 51.1% via a snowball effect.
• Theoretical undercurrent: ICRL. Monea et al. (arXiv:2410.05362) showed LLMs can act as in-context reinforcement learners when given their own past predictions plus a scalar reward. Subsequent work, including "Reward Is Enough" (arXiv:2506.06303) and TR-ICRL (arXiv:2604.00438), pushed this further; TR-ICRL raised Qwen2.5-7B on AIME2024 by 137.59%. All point to the same insight: learning can live in context and inference rather than in weights.
• JitRL's three-pillar loop.
1. *Memory construction:* every interaction is saved as a <state, action, reward> triple; an LLM evaluator assigns step-level rewards to solve credit assignment. 2. *Test-time value estimation:* for a new state s, the agent retrieves k similar past states (N-gram Jaccard), estimates V(s) and Q(s,a), and computes advantage A(s,a). 3. *Logit modulation:* the advantage is added to the base logits before the softmax.
• Closed-form justification. Maximizing expected advantage under the constraint D_KL(pi || pi_base) <= epsilon yields the exponential-tilting policy pi*(a|s) ∝ pi_base(a|s) · exp((1/beta) A(s,a)). Taking logs converts the product to the additive update z'(s,a) = z(s,a) + beta * A(s,a), which is exactly JitRL's rule. The KL term acts as a safety belt preventing language degradation.
• Empirical results. On Jericho, Zork1 rises 35.2 → 52.8 (+50%), Library 18.5 → 28.3 (+53%), Detective 180 → 265 (+47%). On WebArena-Lite, GPT-4o rises 32.5% → 41.2% (+27%), Claude-3.5 28.8% → 38.5% (+34%), Gemini-2.5 25.3% → 34.7% (+37%). JitRL beats WebRL in most domains while cutting cost by 30x or more (~$98 inference vs. ~$3,500 training).
• Engineering note. Structured textual state summaries are more robust than embedding-based retrieval, reinforcing that, for LLMs, natural language is often competitive with vector representations.
• Limitations.
1. Retrieval quality is assumed, not proven; noisy or large memory pools can mislead advantage estimates. 2. Performance drops on genuinely novel tasks with no nearby history, echoing Monea et al.'s warning about exploration fragility and potential model collapse. 3. Highly dynamic UIs with popups can destabilize state representations and reward estimation.
• Outlook. JitRL is not framed as a replacement for gradient RL but as a complementary regime that shifts part of learning from training time to deployment time. It aligns with the broader "test-time scaling" trend exemplified by o1 and DeepSeek-R1, suggesting that extra inference compute can be spent not only on thinking deeper but also on learning better, with formal optimality guarantees.

Core references

1. Li Y., Lin Z., Deng A., et al. *Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates.* ICML 2026. arXiv:2601.18510. 2. Shinn N., Cassano F., Berman E., et al. *Reflexion: Language Agents with Verbal Reinforcement Learning.* NeurIPS 2023. arXiv:2303.11366. 3. Zhao A., Huang D., Xu Q., et al. *ExpeL: LLM Agents Are Experiential Learners.* AAAI 2024. arXiv:2308.10144. 4. Wang Z., Mao J., Fried D., Neubig G. *Agent Workflow Memory.* 2024. arXiv:2409.07429. 5. Qi Z., Liu X., Iong I., et al. *WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning.* ICLR 2025. arXiv:2411.02337. 6. Monea G., Bosselut A., Brantley K., Artzi Y. *LLMs Are In-Context Reinforcement Learners.* 2024. arXiv:2410.05362. 7. Song K., Moeini A., Wang P., et al. *Reward Is Enough: LLMs Are In-Context Reinforcement Learners.* 2026. arXiv:2506.06303. 8. Jiang W., Zuo Y., Zhang Z., et al. *TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning.* 2026. arXiv:2604.00438.