> Paper: Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates > Venue: ICML 2026 Spotlight | Team: National University of Singapore > arXiv: 2601.18510 | TL;DR: Keep model weights frozen throughout; retrieve an experience memory to estimate action advantages and modulate logits directly — no gradient updates, yet the agent keeps improving over its lifetime.
---
Why can't agents just "remember" what they've learned?
Consider an LLM agent trained to book flights. It performs well on training flows, but once deployed, an airline changes its website layout and the agent breaks — its weights were frozen at deployment, so it can't learn the new click sequence.
This is the core dilemma for LLM agents: weights are frozen post-deployment, making continual adaptation hard. The standard answer — continued RL fine-tuning (PPO, GRPO) — faces three obstacles:
1. Prohibitive compute: thousands of dollars per run, H100/H200 clusters running for days 2. Catastrophic forgetting of old environments 3. In-context learning is insufficient: prompts have length limits, and many skills (e.g., "how to bypass this popup on this specific site") can't be verbalized
The paper's core question: can an agent continuously learn new skills after deployment without touching weights or computing gradients? JitRL's answer: yes — store experiences in memory and retrieve them at inference time.
---
Method: a zero-gradient three-step loop
At inference: retrieve similar experiences → estimate action advantages → add them to logits when choosing actions. After each task: evaluate the trajectory → store it in memory → reuse next time.
Step 1: Retrieve
Given a new state, the agent searches memory for the most similar historical states. Memory stores raw (state, action, return) triples — uncompressed experience, unlike EvolveR which distills trajectories into textual principles.
Step 2: Estimate Advantage
- V(s) = average return over historical trajectories whose states are similar to s
- Q(s,a) = average return over similar states where the same action a was taken
- A(s,a) = Q(s,a) − V(s)
- Neighbor search: state similarity is measured as embedding-space distance; the K nearest neighbors are retrieved. Neighbor returns are *realized* returns, not model-estimated ones, so value estimates are noisy — but averaging over enough neighbors works well in practice.
- Theory: under a KL constraint (new policy must stay close to the original), maximizing expected return yields exactly "logits + β × advantage," derived for linear function approximation + softmax policies; the authors argue it transfers to LLMs' discrete token space since the final layer is a large softmax. As memory accumulates, retrieved value estimates converge to true values and the policy converges to the KL-regularized optimum.
- Engineering: white-box models (logprob APIs, open-source models) read candidate token logprobs as logits, add advantages, re-softmax. Black-box models (e.g., some Gemini/Claude versions without probability access) report *verbalized confidence*, converted to approximate logits, then modulated. Both modes work with off-the-shelf APIs.
- JitRL is the strongest training-free method; on the held-out test set it reaches 60.00% vs 46.06% for WebRL.
- Training-free beats gradient fine-tuning — frozen weights + memory retrieval may generalize better than fine-tuning, which overfits the training distribution.
- JitRL's 53.0 is 3.27× GRPO. In extremely sparse state spaces, gradient updates struggle to explore, while memory retrieval directly reuses past successful paths.
- Chen, W., et al. "Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates." ICML 2026 Spotlight. arXiv:2601.18510.
- Related coverage: EvolveR, ProcMEM, GAM
Analogy: at an intersection, the average return of similar past situations is 100 (V=100); the average return when a left turn was taken is 130 (Q=130); so left turn's advantage is +30.
For unseen actions, JitRL applies an optimism-under-uncertainty bonus: the less memory covers an action, the larger the exploration bonus, automatically shifting from exploration to exploitation.
Step 3: Modulate Logits
The advantage is added directly to the model's output logits:
> new logits = original logits + β × advantage
β is a temperature coefficient. Positive-advantage actions get boosted; negative ones suppressed. The paper proves a strong result: this additive rule is the exact closed-form solution to policy optimization under a KL-divergence constraint, preventing abrupt policy shifts.
Closing the loop: evaluation and storage
After each task, an evaluator model performs step-level reward attribution — judging each step's positive/negative contribution to the outcome — then discount-accumulates returns and stores the (state, action, return) triple in memory.
---
Key technical details
---
Results: training-free vs. fine-tuned
WebArena
| Method | Training | Final success rate | |--------|----------|--------------------| | ReAct | zero-shot | 14.41% | | Reflexion | reflection + in-context | 20.93% | | Training-free baseline | training-free | 43.00% | | JitRL | training-free | 51.35% | | WebRL | gradient fine-tuning | 46.06% |
Jericho text games (Zork1)
| Method | Training | Avg score | |--------|----------|-----------| | ReAct | zero-shot | 6.2 | | Reflexion | reflection | 16.2 | | GRPO | gradient updates | 16.2 | | JitRL | training-free | 53.0 |
Cost
| Method | Compute | Estimated cost | |--------|---------|----------------| | WebRL | H200 cluster × 154 hours | ~$9,900 | | JitRL | API calls | ~$290 |
Over 30× cheaper, with no training infrastructure required.
---
Positioning within experience-driven agent evolution
Compared with EvolveR (same "memory-driven evolution" line):
| Dimension | EvolveR | JitRL | |-----------|---------|-------| | Experience storage | compressed textual principles | raw triples, no compression | | Gradient updates | final GRPO training into weights | zero gradients throughout | | Value function | implicit in parameters | explicit in memory | | Retrieval timing | during training | at inference |
Related work: ProcMEM (closest gradient-free relative, also non-parametric advantage estimation, but requires solidifying experiences into callable skills); GAM (similar frozen-weights philosophy, but manages the input side — what goes into context — while JitRL manages the output side — how action probabilities are modulated).
---
Core insights
1. Memory as the value function. V(s) need not be approximated by a neural network: V(s) ≈ average return of memory neighbors. Works well once the agent accumulates enough trajectories. 2. Frozen weights are a feature, not a bug. Memory retrieval preserves the base model's generalization while covering new distributions via experience — outperforming fine-tuning on held-out tasks. For lifelong LLM agents, the answer may be "store memories well" rather than "keep training." 3. The KL constraint matters. The logits+advantage rule is the exact KL-constrained solution, so advantages "fine-tune" rather than "overturn" behavior, protecting the base policy when memory is sparse. 4. Engineering elegance. No model changes, no training infrastructure, white-box and black-box support — plug-and-play deployment. 5. Open problems: memory growth and retrieval efficiency (no eviction/indexing strategies discussed); sensitivity to fixed neighbor count K; potential multi-task interference in a shared memory; and a theoretical gap between linear-approximation assumptions and LLM token spaces.
---