Fine-Tuning Regimes Define Distinct Continual Learning Problems

Forum topic · 小凯 · 2026-04-27

Summary

This arXiv paper (2604.21927) by Paul-Tiberiu Iordache and Elena Burceanu argues that the fine-tuning regime—the subspace of trainable parameters—is itself a critical evaluation variable in continual learning (CL), not just a fixed experimental setting. The authors formalize adaptation as projected optimization over a fixed trainable subspace and show that changing trainable depth alters the effective update signals underlying both task fitting and knowledge retention. Testing this hypothesis in task-incremental CL across five trainable-depth regimes and four standard methods (online EWC, LwF, SI, GEM), on five benchmarks (MNIST, Fashion MNIST, KMNIST, QMNIST, CIFAR-100) with 11 task orders each, they find method rankings are not consistently invariant across regimes. Deeper adaptation regimes correlate with larger update magnitudes, higher forgetting, and a stronger relationship between the two. The results suggest comparative conclusions in CL may depend strongly on the chosen fine-tuning scheme, motivating regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

Overview

Field: Machine Learning Authors: Paul-Tiberiu Iordache, Elena Burceanu Published: 2026-04-23 arXiv: 2604.21927

Abstract

Continual learning (CL) studies how models can sequentially acquire tasks while retaining previously learned knowledge. Despite substantial progress in benchmarking CL methods, comparative evaluations typically keep the fine-tuning regime fixed. This paper argues that the fine-tuning regime, defined by the subspace of trainable parameters, is itself a critical evaluation variable. The authors formalize adaptation as projected optimization over a fixed trainable subspace and show that changing the trainable depth alters the effective update signals on which both current-task fitting and knowledge retention rely. This analysis motivates the hypothesis that method comparisons are not necessarily invariant across regimes.

Key Findings

The hypothesis is examined in task-incremental CL, covering five trainable-depth regimes and four standard methods: online EWC, LwF, SI, and GEM.
Experiments span five benchmark datasets (MNIST, Fashion MNIST, KMNIST, QMNIST, and CIFAR-100) with 11 task orders per dataset.
The relative ranking of methods is not consistently preserved across regimes.
Deeper adaptation regimes are associated with larger update magnitudes, higher forgetting, and a stronger relationship between the two.

Implications

Comparative conclusions in continual learning may depend strongly on the chosen fine-tuning scheme. The authors advocate regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.

---

*Auto-collected on 2026-04-27.*