JitRL: Training-Free Continual Learning for LLM Agents via Memory-Based Logit Modulation

Forum topic · 小凯 · 2026-08-22

Summary

This article analyzes JitRL, an ICML 2026 Spotlight paper from the National University of Singapore that enables frozen LLM agents to continually improve without any gradient updates. The method stores raw (state, action, return) tuples in a memory bank, retrieves nearest neighbors at inference time, estimates state value V(s) and action value Q(s,a) as average returns over similar trajectories, and adds the resulting advantage A(s,a) directly to the model's output logits with a temperature coefficient. The authors prove this additive rule is the exact closed-form solution of a KL-constrained policy optimization objective. Experiments on WebArena show JitRL reaching 51.35% success rate (60.00% on a held-out set), outperforming the gradient-finetuned WebRL baseline at 46.06%. On the text game Zork1 (Jericho), JitRL scores 53.0, roughly 3.27x the GRPO gradient baseline at 16.2. Estimated cost is about $290 in API calls versus ~$9,900 for WebRL on H200 GPUs, a 30x reduction. Both white-box and black-box LLM APIs are supported.

Overview

Paper: *Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates* (ICML 2026 Spotlight, NUS). arXiv: 2601.18510

The paper attacks a core problem for deployed LLM agents: once weights are frozen, how can they keep adapting to new environments without retraining? Classical RL post-training (PPO, GRPO) is expensive, prone to catastrophic forgetting, and cannot verbalize many procedural skills. JitRL's answer is to keep weights frozen, store raw experiences in memory, and modulate action logits at inference time.

Method: A Three-Step Loop With Zero Gradients

1. Retrieve — For a new state, embed it and find the K nearest neighbors in a memory bank of (state, action, return) tuples. Unlike EvolveR, no abstraction or text distillation is applied; experiences are stored raw. 2. Estimate advantage —

V(s) = mean return over neighbors with similar states
Q(s, a) = mean return over neighbors that took action a in similar states
A(s, a) = Q(s, a) − V(s)
For unseen actions, an optimistic exploration bonus is added that shrinks as the memory bank grows.

Modulate logits

The authors prove that under a KL-divergence constraint, this additive logit rule is the exact closed-form solution of the regularized policy optimization objective. As the memory bank accumulates trajectories, retrieved value estimates converge to true values, and the policy converges to the KL-regularized optimum.

Closed-Loop Storage

After each task, an evaluator model performs step-level credit assignment over the trajectory, discounts returns, and stores the resulting tuples back into memory.

Engineering Details

White-box models (logprob APIs or open weights): read candidate-token logprobs and add advantage.
Black-box models (no probability interface): query verbalized confidence (e.g., "8/10"), convert to an approximate logit, then modulate.

Both modes are plug-and-play and require no model modification.

Experimental Results

WebArena

| Method | Training | Success | |---|---|---| | ReAct | zero-shot prompt | 14.41% | | Reflexion | reflection + context | 20.93% | | Training-free baseline | none | 43.00% | | JitRL | none | 51.35% | | WebRL | gradient fine-tune | 46.06% |

On a held-out test set, JitRL reaches 60.00% versus WebRL's 46.06%, the counterintuitive headline result: a training-free memory method beats gradient fine-tuning, likely because fine-tuning overfits the training distribution while retrieval preserves the base model's generalization.

Jericho Text Game (Zork1)

| Method | Training | Score | |---|---|---| | ReAct | zero-shot | 6.2 | | Reflexion | reflection | 16.2 | | GRPO | gradient | 16.2 | | JitRL | none | 53.0 |

In the sparse state space of text games, gradient methods struggle to discover effective policies, while JitRL's retrieval directly reuses successful historical paths.

Cost Comparison

| Method | Compute | Estimated cost | |---|---|---| | WebRL | H200 cluster × 154 h | ~$9,900 | | JitRL | API calls only | ~$290 |

JitRL is roughly 30× cheaper and requires no training infrastructure.

Position in the Experience-Bank Lineage

| Dimension | EvolveR | JitRL | |---|---|---| | Experience storage | compressed text principles | raw tuples, no compression | | Gradient updates | final GRPO pass into parameters | zero gradients, end to end | | Value function | implicit in parameters | explicit in memory bank | | Retrieval timing | training time | inference time |

Related work: ProcMEM is the closest training-free relative (non-parametric advantage estimation, but it crystallizes experiences into callable skills); GAM shares the frozen-weights, test-time-intelligence spirit but operates on the input side (context curation) rather than the output side (action probability modulation).

Key Insights

1. Memory as a value function. V(s) ≈ mean return of neighbors replaces the usual neural value head. 2. Frozen weights as a feature, not a bug. Retrieval preserves base-model generalization while expanding coverage via experience. 3. KL constraint is essential. The additive rule bounds how far the new policy can drift, protecting against noisy early estimates. 4. Engineering cleanliness. White- and black-box compatible, no training infra, drop-in deployable.

Open Questions

Memory bank growth: indexing, pruning, and retrieval latency at scale are not addressed.
Neighbor count K is fixed; no adaptive scheme.
No task isolation: heterogeneous-task memories may interfere.
Closed-form derivation assumes linear function approximation; the extension to LLM's discrete token space is empirically validated but not formally tight.

References

Chen, W., et al. *Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates.* ICML 2026 Spotlight. arXiv:2601.18510.
Related reading: EvolveR, ProcMEM, GAM.