Paper Overview
Field: Machine Learning Authors: Dominik Żurek, Kamil Faber, Marcin Pietron, et al. Published: 2026-04-29 arXiv: 2504.21087
Summary
Continual offline reinforcement learning (CORL) aims to learn a sequence of tasks from datasets collected over time while preserving performance on previously learned tasks. This setting corresponds to domains where new tasks arise over time, but adapting the model in live environment interactions is expensive, risky, or impossible. However, CORL inherits the dual difficulty of offline reinforcement learning and adapting while preventing catastrophic forgetting.
Replay-based continual learning approaches remain a strong baseline but incur memory overhead and suffer from distribution mismatch between replayed samples and newly learned policies. Meanwhile, architectural continual learning methods have shown strong potential in supervised learning but remain underexplored in CORL.
This paper proposes TSN-Affinity, a novel CORL method based on TinySubNetworks and the Decision Transformer. The method achieves task-specific parameterization and controlled knowledge sharing through an RL-aware reuse policy, routing tasks according to action compatibility and latent similarity.
Evaluation and Results
The method was evaluated on Atari game benchmarks and simulations of Franka Emika Panda robotic arm manipulation tasks, covering both discrete and continuous control. Results show that:
- Sparse subnetworks exhibit strong retention capabilities
- Routing further improves multi-task performance
- Similarity-guided architectural reuse is a strong, viable alternative to replay-based strategies in CORL settings