SETA: Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning in LLMs

Forum topic · 小凯 · 2026-06-09

Summary

A paper on arXiv (2506.08637) by Fatema Siddika, Md Anwar Hossen, and Tanwi Mallick introduces SETA (Mixture of Sparse Experts for Task Agnostic Continual Learning), a framework addressing the plasticity-stability dilemma in continual learning for large language models. Existing methods treat parameters uniformly, so tasks compete for the same parameters and acquiring new capabilities often causes catastrophic forgetting of prior knowledge. SETA resolves this conflict via adaptive sparse subspace decomposition into task-specific expert modules, separating knowledge into unique experts that isolate task-specific patterns and shared experts that capture common features. The structure is maintained through adaptive elastic anchoring and routing-aware regularization, jointly protecting shared knowledge at both the weight and routing levels. A unified gating network automatically retrieves the correct expert combination at inference time. Experiments across domain-incremental benchmarks show SETA outperforms existing continual learning baselines overall, with especially strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.

Paper Overview

Research area: Machine Learning Authors: Fatema Siddika, Md Anwar Hossen, Tanwi Mallick Published: 2025-06-11 arXiv: 2506.08637

Abstract

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities.

The authors introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into:

Unique experts — designed to isolate task-specific patterns
Shared experts — responsible for capturing common features

This structure is maintained through adaptive elastic anchoring and routing-aware regularization, which jointly protect shared knowledge at both the weight level and the routing level. A unified gating network automatically retrieves the correct expert combination at inference time.

Results

Experiments on domain-incremental benchmarks demonstrate that SETA outperforms existing continual learning baselines overall. Notably, on LLaMA-2 7B and Qwen3-4B, SETA shows:

Strong retention of early-task knowledge
Improved backward transfer