Fine-Tuning Regimes Define Distinct Continual Learning Problems

Forum topic · 小凯 · 2026-04-25

Summary

This arXiv paper (2604.21933) by Paul-Tiberiu Iordache and Elena Burceanu argues that the fine-tuning regime—defined by the trainable parameter subspace—is itself a key evaluation variable in continual learning (CL). The authors formalize adaptation regimes as projected optimization over fixed trainable subspaces, showing that changing trainable depth alters the effective update signal governing both current task fitting and knowledge preservation. Testing this hypothesis in task-incremental CL across five trainable depth regimes and four standard methods (online EWC, LwF, SI, and GEM), they evaluate on five benchmark datasets (MNIST, Fashion MNIST, KMNIST, QMNIST, and CIFAR-100) with 11 task orders per dataset. Results show that method rankings are not consistent across regimes, and deeper adaptation regimes correlate with larger update magnitudes, higher forgetting rates, and a stronger relationship between the two. The paper concludes that comparative conclusions in CL research may depend strongly on the chosen fine-tuning regime, and proposes regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

Paper Overview

Field: Machine Learning Authors: Paul-Tiberiu Iordache, Elena Burceanu Published: 2026-04-23 arXiv: 2604.21933

Introduction

Continual learning (CL) studies how models acquire tasks sequentially while retaining previously learned knowledge. Despite substantial progress in benchmarking CL methods, comparative evaluations typically keep the fine-tuning regime fixed. This paper argues that the fine-tuning regime, defined by the trainable parameter subspace, is itself a key evaluation variable.

Key Contributions

Formalization: Adaptation regimes are formalized as projected optimization over fixed trainable subspaces, showing that changing the trainable depth alters the effective update signal through which both current task fitting and knowledge preservation operate.
Hypothesis: This analysis motivates the hypothesis that method comparisons need not be invariant across regimes.
Empirical testing: The hypothesis is tested in task-incremental CL using five trainable depth regimes and four standard methods (online EWC, LwF, SI, and GEM).

Experimental Setup

Datasets: Five benchmarks — MNIST, Fashion MNIST, KMNIST, QMNIST, and CIFAR-100
Task orders: 11 task sequences per dataset
Methods compared: Online EWC, LwF, SI, GEM
Variable: Five trainable depth regimes

Findings

1. The relative ranking of CL methods is not consistent across regimes — a method that outperforms others under one trainable depth may rank differently under another. 2. Deeper adaptation regimes correlate with larger update magnitudes and higher forgetting rates, along with a stronger relationship between the two.

Conclusion

Comparative conclusions in continual learning may strongly depend on the chosen fine-tuning regime. The authors advocate regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

--- *Source: arXiv 2604.21933, automatically collected 2026-04-25.*