Learning, Fast and Slow: Towards LLMs That Adapt Continually

Forum topic · 小凯 · 2026-05-14

Summary

This arXiv paper (2605.12484) introduces a fast-slow learning (FST) framework for large language models that combines parameter updates with context optimization. Standard RL-based fine-tuning forces task-specific information into weights, risking catastrophic forgetting and loss of plasticity, while in-context learning adapts cheaply but often underperforms weight updates. FST treats model parameters as 'slow' weights and optimized context as 'fast' weights, learning task-specific knowledge in context while keeping weights closer to the base model. Experiments show FST achieves 3x better sample efficiency than RL-only learning on reasoning tasks with higher performance asymptotes, up to 70% lower KL divergence from the base LLM, reduced catastrophic forgetting, and preserved plasticity for adapting to subsequent tasks. In continual learning settings, FST keeps acquiring new tasks where parameter-only RL stagnates.

Overview

Field: Machine Learning Authors: Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri Published: 2026-05-12 arXiv: 2605.12484

Abstract (translated)

Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs System 2).

To this end, the authors introduce a fast-slow learning framework for LLMs, with model parameters serving as slow weights and optimized context as fast weights. The fast "weights" can be learned from textual feedback to absorb task-specific information, while allowing the slow weights to stay closer to the base model and preserve general reasoning behavior.

Key Findings

Sample efficiency: Fast-slow training (FST) is 3x more sample-efficient than slow-only learning (RL) on reasoning tasks, while consistently reaching higher performance asymptotes.
Reduced drift: FST-trained models show up to 70% lower KL divergence from the base LLM, resulting in less catastrophic forgetting compared to RL training.
Preserved plasticity: After training on one task, FST models adapt more effectively to subsequent tasks than parameter-only trained models.
Continual learning: In continual learning scenarios, FST keeps acquiring each new task, whereas parameter-only RL stagnates.