Latent Dynamics for Full Body Avatar Animation: Pose-Conditioned 3D Gaussian Avatars with Learned Dynamic Residual Latents

Forum topic · 小凯 · 2026-05-22

Summary

A CVPR-style research paper (arXiv:2505.15980) by Shichong Peng, Chengxiang Yin, and Fei Jiang, published May 20, 2025, introduces a latent dynamics approach for animating full-body avatars with loose clothing. Pose-driven neural rendering avatars struggle because garments deform based on history, inertia, and contact—states a pose alone cannot explain. The proposed method augments a pose-conditioned 3D Gaussian avatar with dynamic residual latent variables that capture appearance and geometric changes beyond the driving signal, decoded via a transformer-based decoder. At inference, a learned latent dynamics model evolves the residual latent from a short pose history and the previous latent state, decomposing each update into driving, restoring, and dissipating forces. This yields temporally consistent, history-dependent rollouts with negligible extra cost, enables diverse yet plausible motion trajectories from different initial conditions, and exposes controllable parameters such as stiffness. Evaluated on nine motion-capture sequences featuring everyday movements in loose clothing, quantitative metrics and a perceptual user study show animation quality superior to recent data-driven baselines—without requiring garment templates or test-time physics simulators.

Paper Overview

Field: Computer Vision
Authors: Shichong Peng, Chengxiang Yin, Fei Jiang
Published: 2025-05-20
arXiv: 2505.15980

Problem

Pose-driven full-body avatars built on neural rendering produce high-quality novel views of a captured subject. Yet loose clothing and other dynamic elements deform in ways pose alone cannot explain: the same pose can correspond to many different states, because their motion depends on history, inertia, and contact.

Existing approaches have limitations:

Explicit simulation and layered-garment methods can model such dynamics, but they require either a dedicated garment template (which raw multi-view capture does not naturally provide) or a test-time physics simulator with non-trivial runtime cost.
Data-driven clothing avatars avoid explicit garment layers by adding an auxiliary latent for variation beyond pose. However, at inference they fix the latent, regress it from pose, or retrieve it from training data—without explicitly modeling how the latent evolves with its own dynamics. Moreover, existing architectures often fail to capture fine-grained details in everyday loose-clothing motion, producing blurry renderings and temporal artifacts.

Method

The authors augment a pose-conditioned 3D Gaussian avatar with:

A transformer-based decoder
Dynamic residual latent variables that capture temporal appearance and geometric changes beyond the driving pose signal

At inference, a learned latent dynamics model evolves the residual latent from a short pose history and the previous latent state. Each update is decomposed into:

1. Driving force 2. Restoring force 3. Dissipating force

This decomposition produces temporally consistent, history-dependent rollouts at negligible additional cost.

Results

Different initial conditions produce diverse but plausible motion trajectories
The force decomposition exposes controllable parameters such as stiffness
On nine diverse motion-capture sequences of everyday movements in loose clothing, quantitative metrics and a perceptual user study show animation quality outperforming recent data-driven baselines