Fine-Tuning Regimes Define Distinct Continual Learning Problems

Forum topic · 小凯 · 2026-04-25

Summary

This arXiv paper (2604.21933) by Paul-Tiberiu Iordache and Elena Burceanu argues that the fine-tuning regime—defined by the trainable parameter subspace—is itself a key evaluation variable in continual learning (CL), not just a fixed background setting. The authors formalize adaptation regimes as projected optimization over fixed trainable subspaces and show that changing the trainable depth alters the effective update signal governing both task fitting and knowledge retention. Testing four standard CL methods (online EWC, LwF, SI, and GEM) across five trainable-depth regimes, five benchmark datasets (MNIST, Fashion MNIST, KMNIST, QMNIST, CIFAR-100), and 11 task orders per dataset, they find that relative method rankings are not consistent across regimes. Deeper adaptation regimes correlate with larger update magnitudes, higher forgetting, and a stronger relationship between the two. The results suggest that comparative conclusions in CL research depend strongly on the chosen fine-tuning regime, motivating regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

Paper Overview

Field: Machine Learning
Authors: Paul-Tiberiu Iordache, Elena Burceanu
Published: 2026-04-23
arXiv: 2604.21933

Abstract

Continual learning (CL) studies how models acquire tasks sequentially while retaining previously learned knowledge. Despite substantial progress in benchmarking CL methods, comparative evaluations typically keep the fine-tuning regime fixed. In this paper, the authors argue that the fine-tuning regime, defined by the trainable parameter subspace, is itself a key evaluation variable.

Key Points

The paper formalizes adaptation regimes as projected optimization over fixed trainable subspaces, showing that changing the trainable depth alters the effective update signal through which both current task fitting and knowledge preservation operate.
This analysis motivates the hypothesis that method comparisons need not be invariant across regimes.
The hypothesis is tested in task-incremental CL using five trainable depth regimes and four standard methods: online EWC, LwF, SI, and GEM.
Experiments cover five benchmark datasets (MNIST, Fashion MNIST, KMNIST, QMNIST, and CIFAR-100), with 11 task orders per dataset.
Findings: the relative ranking of methods is not consistent across regimes.
Deeper adaptation regimes are associated with larger update magnitudes, higher forgetting rates, and a stronger relationship between the two.

Conclusion

Comparative conclusions in continual learning may depend strongly on the chosen fine-tuning regime. The authors propose regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

---

*Auto-collected on 2026-04-25*