T2-Z: Test-Time Optimization of Environment Latents for Robotic World Models

Forum topic · 小凯 · 2026-10-17

Summary

T2-Z is a method for test-time optimization of environment latents in robotic world models, addressing factors that are neither directly observable nor explicitly specified—such as friction, mass, or human-defined environmental rules. The approach augments a robot world model with a learnable environment latent that conditions world predictions, capturing environmental variation without any supervision of what factors exist or their values. Training alternates between a world-model optimization stage and an environment-latent optimization stage. At test time, the world model stays frozen while freshly initialized environment latents are optimized from one or more test-time interactions, adapting the model to unseen environment configurations. Across 7 simulated environments and 4 real robot tasks, T2-Z reduces object prediction error from 15.20 to 7.00 and improves action success rate from 33.73% to 72.96%. Ablations show T2-Z can efficiently infer environment latents from a single test-time interaction trajectory. Paper: arXiv 2610.22364.

Paper Overview

Field: Computer Vision (CV)

Authors: Chenyuan Zhou, Yining Hong, Huang Huang, Yunfan Jiang, Wenlong Huang, David Held, Vitor Campagnolo Guizilini, Yue Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Yejin Choi

arXiv: 2610.22364

Abstract

Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, from physical properties such as friction and mass to human-defined environmental rules. The authors introduce T2-Z for test-time optimization of environment latents, which augments a robotic world model with a learnable environment latent to condition world predictions. This compact latent captures underlying environmental variation without supervision of what environmental factors exist or what values they take.

Method

Training: T2-Z alternates between two stages — a world-model optimization stage that fixes the latents and learns general action-conditioned dynamics across environments, and an environment-latent optimization stage.
Test time: The world model remains frozen, while newly initialized environment latents are optimized from one or more test-time interactions, allowing the model to adapt to unseen environment configurations.

Results

Evaluated on 7 simulated environments and 4 real robot tasks
Object prediction error reduced from 15.20 → 7.00
Action success rate improved from 33.73% → 72.96%
Ablation studies show T2-Z can efficiently infer environment latents from a single test-time interaction trajectory

---

*Auto-collected on 2026-10-17*