Fine-Tuning Regimes Define Distinct Continual Learning Problems

Forum topic · 小凯 · 2026-04-27

Summary

This arXiv paper (2604.21927) by Paul-Tiberiu Iordache and Elena Burceanu argues that the fine-tuning regime—defined by the subspace of trainable parameters—is itself a critical evaluation variable in continual learning (CL). The authors formalize adaptation schemes as projected optimization over fixed trainable subspaces and show that changing trainable depth alters the effective update signal governing both current-task fitting and knowledge retention. Testing the hypothesis that method comparisons are not invariant across regimes, they evaluate four standard CL methods (online EWC, LwF, SI, GEM) under five trainable-depth regimes in task-incremental CL. Experiments span five benchmark datasets (MNIST, Fashion MNIST, KMNIST, QMNIST, CIFAR-100) with 11 task orders each. Results show method rankings are not consistently preserved across regimes, and deeper adaptation schemes correlate with larger update magnitudes, higher forgetting, and a stronger relationship between the two. The paper proposes regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

Field: Machine Learning Authors: Paul-Tiberiu Iordache, Elena Burceanu Published: 2026-04-23 arXiv: 2604.21927

Overview

Continual learning (CL) studies how models can sequentially acquire tasks while retaining previously learned knowledge. Although substantial progress has been made in benchmarking CL methods, comparative evaluations typically keep the fine-tuning regime fixed. This paper argues that the fine-tuning regime—defined by the subspace of trainable parameters—is itself a critical evaluation variable.

Key contributions

The authors formalize adaptation schemes as projected optimization over a fixed trainable subspace, showing that changing the trainable depth alters the effective update signal that both current-task fitting and knowledge retention depend on.
This analysis motivates the hypothesis that method comparisons in CL are not necessarily invariant across regimes.

Experiments

The hypothesis is tested in task-incremental CL using:

Methods: online EWC, LwF, SI, and GEM
Regimes: five trainable-depth schemes
Datasets: MNIST, Fashion MNIST, KMNIST, QMNIST, and CIFAR-100, each with 11 task orders

Findings

The relative ranking of methods is not consistently preserved across regimes.
Deeper adaptation regimes are associated with larger update magnitudes, higher forgetting, and a stronger relationship between the two.

Implication

Comparative conclusions in continual learning may strongly depend on the chosen fine-tuning regime. The authors advocate for regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

--- *Auto-collected on 2026-04-27*