Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning (SETA)

Forum topic · 小凯 · 2026-06-09

Summary

SETA (Mixture of Sparse Experts for Task Agnostic Continual Learning) is a framework for continual learning in large language models that addresses the plasticity-stability dilemma, where learning new capabilities often causes catastrophic forgetting of prior knowledge. Unlike standard methods that treat parameters uniformly, SETA performs adaptive sparse subspace decomposition into expert modules, separating knowledge into unique experts that isolate task-specific patterns and shared experts that capture common features. The structure is maintained through adaptive elastic anchoring and routing-aware regularization, jointly protecting shared knowledge at both the weight and routing levels, while a unified gating network automatically retrieves the right expert combination at inference. Experiments across domain-incremental benchmarks show SETA outperforms existing continual learning baselines, with notably strong early-task knowledge retention and improved backward transfer on LLaMA-2 7B and Qwen3-4B.

Paper Overview

Field: Machine Learning Authors: Fatema Siddika, Md Anwar Hossen, Tanwi Mallick Published: 2025-06-11 arXiv: 2506.08637

Abstract

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities.

The authors introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into:

Unique experts — designed to isolate task-specific patterns
Shared experts — responsible for capturing common features

This structure is maintained through adaptive elastic anchoring and routing-aware regularization, which jointly protect shared knowledge at both the weight and routing levels, and enable a unified gating network to automatically retrieve the correct combination of experts at inference time.

Results

Experiments across domain-incremental benchmarks show that SETA outperforms existing continual learning baselines overall, demonstrating particularly strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.

---

*Auto-collected on 2026-06-09.*