Continual Learning Shouldn't Just Avoid Forgetting—It Should Learn What's Invariant

Forum topic · 小凯 · 2026-05-18

Summary

A paper by Janetzky, Schlagenhauf, and Feuerriegel (LMU Munich, ICML 2026) argues that continual learning (CL) methods optimize only in-domain performance, causing models to learn domain-specific shortcuts—e.g., recognizing scanner brands in medical imaging or factory lighting in manufacturing rather than true pathology or defects—leading to sharp performance drops on unseen domains. The authors propose continual learning of domain-invariant representations, grounded in the idea that causal mechanisms remain constant across domains while noise (equipment, lighting, patient distribution) varies. Their method combines replay-based training with sequential invariance alignment: beyond preventing forgetting via a replay buffer, an alignment objective ensures newly learned representations share the same invariant structure as those from earlier domains. Experiments across six datasets spanning vision, medical imaging, manufacturing, and ecology show consistent gains over CL baselines on unseen target domains. Ablations reveal that simply chaining off-the-shelf domain-invariant representation learning into a CL pipeline is insufficient—invariance must be actively maintained within the continual learning loop. Open questions include handling domains whose causal structures differ (e.g., changed clinical guidelines), whether causal mechanisms are verifiable from data, and the computational overhead of alignment.

Continual learning's core goal: a model is trained sequentially across multiple domains, learning new knowledge without forgetting the old. But in a paper at ICML 2026, Janetzky, Schlagenhauf, and Feuerriegel (LMU Munich) point out that existing methods miss a key problem: they only optimize in-domain performance, causing models to learn domain-specific shortcuts—in medical imaging, a model may learn to recognize the brand of the scanning device rather than genuine pathology; in manufacturing, it may learn to recognize the lighting conditions of a factory floor rather than product defects.

Such models suffer catastrophic performance drops when moved to new domains.

Key points

The paper proposes a class of methods for continual learning of domain-invariant representations.
Core idea: causal relationships are invariant across domains—the physiological mechanism causing a disease is the same at hospital A and hospital B, while noise (equipment, lighting, patient distribution) changes. If the causal structure can be learned and preserved during continual learning, the model can truly generalize to new domains.
Method: combines replay-based training with sequential invariance alignment. Each time the model is exposed to a new domain, a replay buffer is used to prevent forgetting old tasks, and an alignment objective ensures that newly learned representations share the same invariant structure as representations from earlier domains. The invariant structure is tracked and consolidated throughout the learning sequence.
Results: across six datasets—vision, medical, manufacturing, ecology—the proposed method consistently outperforms existing CL baselines on unseen target domains.
Ablation: simply chaining an off-the-shelf domain-invariant representation learning method into a CL pipeline gives limited benefit—invariance must be actively maintained within the continual learning loop; it cannot be layered on afterward.

Open questions

Tuning the strength of invariance alignment: if the causal structures of two domains are not fully consistent (e.g., clinical guidelines have changed), forced alignment can actually be harmful.
Are the "causal mechanisms" mentioned in the paper an assumption, or can they be confirmed from data?
Computational overhead: how much extra training time does each alignment step require?

References

1. Janetzky, P., Schlagenhauf, T., & Feuerriegel, S. (2026). *Continual Learning of Domain-Invariant Representations*. arXiv:2605.15775 [cs.LG]. 2. Schölkopf, B., et al. (2021). *Towards Causal Representation Learning*. Proceedings of the IEEE. 3. Kirkpatrick, J., et al. (2017). *Overcoming Catastrophic Forgetting in Neural Networks*. PNAS.