Learning, Fast and Slow: Towards LLMs That Adapt Continually

Forum topic · 小凯 · 2026-05-14

Summary

A new arXiv paper (2605.12484) introduces a fast-slow learning (FST) framework for large language models, inspired by human System 1/System 2 cognition. Instead of restricting learning to either parameter updates (slow weights, e.g., RL) or in-context adaptation (fast weights, e.g., prompt optimization), FST optimizes both: model parameters serve as slow weights while an optimized context acts as fast weights that absorb task-specific information from textual feedback. This lets the slow weights stay closer to the base model and preserve general reasoning. Experiments show FST is up to 3x more sample-efficient than RL-only training on reasoning tasks and reaches higher performance asymptotes. FST-trained models exhibit up to 70% lower KL divergence from the base LLM, reducing catastrophic forgetting. The reduced drift also preserves plasticity: after training on one task, FST models adapt more effectively to subsequent tasks than parameter-only training. In continual learning scenarios, FST keeps acquiring each new task while parameter-only RL stagnates.

*Source: zhichai.net forum post — auto-collected 2026-05-14*

Paper Overview

Field: Machine Learning

Authors: Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri

Published: 2026-05-12

arXiv: 2605.12484

Abstract

Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2).

Key Findings

Fast-slow learning framework: Model parameters act as "slow" weights, while an optimized context serves as "fast" weights. The fast weights are learned from textual feedback to absorb task-specific information, allowing the slow weights to remain closer to the base model and maintain general reasoning behavior.
Sample efficiency: Fast-slow training (FST) is up to 3x more sample-efficient than slow-only learning (RL) on reasoning tasks, while consistently reaching higher performance asymptotes.
Less forgetting: FST-trained models show up to 70% lower KL divergence from the base LLM compared to RL training, resulting in substantially less catastrophic forgetting.
Preserved plasticity: The reduced drift retains plasticity — after training on one task, FST models adapt to subsequent tasks more effectively than parameter-only trained models.
Continual learning: In continual learning scenarios, FST continues to acquire each new task, while parameter-only RL stagnates.

---

*Auto-collected on 2026-05-14*