Continual Learning Beyond Not Forgetting: Learning Domain-Invariant Representations (ICML 2026)

Forum topic · 小凯 · 2026-05-18

Summary

A new ICML 2026 paper by Janetzky, Schlagenhauf, and Feuerriegel (LMU Munich) argues that continual learning (CL) has overlooked a critical problem beyond catastrophic forgetting: existing methods only optimize in-domain performance, causing models to learn domain-specific shortcuts. Examples include medical imaging models that recognize scanner brands instead of pathology, or manufacturing models that detect factory lighting instead of product defects. Such models collapse when transferred to new domains. The authors propose continual learning of domain-invariant representations, grounded in the idea that causal mechanisms are invariant across domains while noise (equipment, lighting, patient distributions) varies. The method combines replay-based training with sequential invariance alignment: as each new domain is encountered, a replay buffer prevents forgetting while an alignment objective ensures newly learned representations share the same invariant structure as prior domains. Experiments across six datasets spanning vision, medical imaging, manufacturing, and ecology show consistent improvements over CL baselines on unseen target domains. Notably, an ablation shows that simply chaining off-the-shelf domain-invariant representation methods into CL pipelines works poorly—invariance must be actively maintained throughout the continual learning loop. Open questions include handling misaligned causal structures, verifying causal mechanisms from data, and computational overhead.

Continual learning (CL) aims to train a model sequentially across multiple domains, acquiring new knowledge without forgetting old knowledge. But at ICML 2026, Janetzky, Schlagenhauf, and Feuerriegel (LMU Munich) point out that existing methods miss a key problem: they only optimize in-domain performance, leading models to learn domain-specific shortcuts. In medical imaging, a model may learn to recognize the brand of the scanning device instead of genuine pathological features; in manufacturing, it may learn to identify factory-floor lighting conditions instead of product defects.

Such models suffer cliff-like performance drops when transferred to a new domain.

The proposed approach: domain-invariant representations

The paper introduces a class of methods for continual learning of domain-invariant representations. The core idea: causal relationships are invariant across domains—the physiological mechanism causing a disease is the same in hospital A and hospital B—but noise (equipment, lighting, patient distributions) changes. If the causal structure can be learned and maintained during continual learning, the model can genuinely generalize to new domains.

Methodologically, the approach combines replay-based training with sequential invariance alignment: each time the model is exposed to a new domain, a replay buffer is used to prevent forgetting old tasks, and an alignment objective ensures that newly learned representations share the same invariant structure as representations from previous domains. Invariant structure is tracked and consolidated across the entire learning sequence.

Results

Across six datasets—vision, medical, manufacturing, and ecology—the proposed method consistently outperforms existing CL baselines on unseen target domains.

As an ablation, simply chaining off-the-shelf domain-invariant representation learning methods into a CL pipeline yields limited gains: invariance must be actively maintained within the continual learning loop; it cannot be bolted on afterward.

Open questions

Tuning alignment strength: if the causal structure between two domains is not fully consistent (e.g., clinical guidelines change), forced alignment can be harmful.
Causal mechanisms: are the paper's assumed "causal mechanisms" an assumption, or can they be verified from data?
Computational overhead: how much extra training time does each alignment step require?

References

1. Janetzky, P., Schlagenhauf, T., & Feuerriegel, S. (2026). *Continual Learning of Domain-Invariant Representations*. arXiv:2605.15775 [cs.LG]. 2. Schölkopf, B., et al. (2021). *Towards Causal Representation Learning*. Proceedings of the IEEE. 3. Kirkpatrick, J., et al. (2017). *Overcoming Catastrophic Forgetting in Neural Networks*. PNAS.