What Should World Models Forget? Stratified Retention for Continual World Models

Forum topic · 小凯 · 2026-10-06

Summary

This position paper by Nishit Anand, Ramani Duraiswami, and Dinesh Manocha (arXiv:2610.03713) challenges the standard continual learning assumption that performance degradation on previously seen data signals failure. The authors argue that world models predict a changing environment, so knowledge that was accurate when acquired may become false, and discarding it is required behavior rather than a defect. Because world models also encode knowledge that must never be revised, they propose retention stratified by invariance timescale: invariants such as physical laws and object permanence should never change, while instance-level facts should be updated as soon as the environment changes. Standard forgetting metrics cannot distinguish a world model that correctly updated stale knowledge from one suffering catastrophic forgetting—they rank frozen models highest—and existing physics-reasoning benchmarks evaluate only frozen checkpoints. The paper proposes differentiated retention, jointly reporting invariant regression tests on adaptation streams alongside correction latency, without aggregation.

Paper Overview

Field: Computer Vision Authors: Nishit Anand, Ramani Duraiswami, Dinesh Manocha Published: 2026-10-02 arXiv: 2610.03713

Summary (Translated)

Continual learning treats degradation on previously seen data as evidence of failure—a convention inherited from settings with a stationary prediction target, where a correct label remains correct indefinitely. World models do not satisfy this condition. Their prediction target is the environment, which changes, so knowledge that was accurate when acquired may later become false, and discarding it is required behavior rather than a defect.

Non-stationary ground truth is well studied in the concept drift literature and in the temporal factuality of language models, but has not been formulated for world models, which are distinctive in that they also encode knowledge that must never be revised. The authors argue that continual world models require retention stratified by invariance timescale, separating invariants—such as physical laws and object permanence, which must never be modified—from instance-level facts, which should be updated as soon as the environment changes.

Standard forgetting metrics cannot distinguish a world model that correctly updated stale knowledge from one suffering catastrophic forgetting, and therefore rank frozen models highest; existing physics-reasoning benchmarks evaluate only frozen checkpoints. The paper proposes differentiated retention: jointly reporting invariant regression tests on adaptation streams alongside correction latency, without aggregation.

Key Ideas

The continual learning "forgetting = failure" assumption breaks down when the prediction target (the environment) is non-stationary.
Knowledge should be retained according to its invariance timescale: invariants (physics, object permanence) are protected; instance-level facts are actively refreshed.
Aggregated forgetting metrics conflate beneficial updates with catastrophic forgetting, biasing evaluations toward frozen models.
Evaluation should separately track invariant regression (stability) and correction latency (adaptability) on adaptation streams.

*Auto-collected on 2026-10-06*