4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction

Forum topic · 小凯 · 2026-10-08

Summary

4D-HOF is a feed-forward framework for reconstructing 4D hand-object interactions, presented in arXiv paper 2610.08782. Existing 4D hand-object reconstruction methods typically rely on costly per-sequence optimization, while generative approaches synthesize interactions from random noise, which can produce unstable predictions. 4D-HOF instead reconstructs interactions from coarse but informative estimates produced by vision foundation models. The authors train a conditional flow matching model that transports foundation-model-derived hand-object states toward an interaction manifold, correcting errors in translation, rotation, and alignment in a feed-forward manner. A key advantage of this generative formulation is that test-time guidance is naturally built into the transport process: physical interaction constraints and observed 2D evidence directly guide the evolving generative states, so refinement occurs as part of generation itself rather than as a separate post-hoc optimization step. Trained on diverse datasets, 4D-HOF generalizes robustly to challenging real-world scenes and achieves state-of-the-art performance on out-of-domain benchmarks with more stable and accurate 4D reconstructions.

Paper Overview

Field: Computer Vision
Authors: Shiqi Li, Sean Cho, Yijie Li, Fengzhi Guo, Bowen Wen, Cheng Zhang
Published: 2026-10-06
arXiv: 2610.08782

Abstract

Existing methods for 4D hand-object reconstruction often rely on costly per-sequence optimization, while generative approaches typically synthesize interactions from random noise, which can lead to unstable interaction prediction. We introduce 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions from coarse but informative estimates produced by vision foundation models.

Concretely, we learn a conditional flow matching model that transports foundation-model-derived hand-object states toward an interaction manifold, allowing the model to correct errors in translation, rotation, and alignment in a feed-forward manner.

A key advantage of our generative formulation is that it naturally enables test-time guidance within the transport process. Rather than applying a separate post-hoc refinement step after modeling, physical interaction constraints and observed 2D evidence directly guide the evolving generative states, so reconstruction is refined as part of the generative process itself.

By training the generative model on diverse datasets, 4D-HOF generalizes robustly to challenging real-world scenarios. Experiments on out-of-domain benchmarks show that 4D-HOF achieves state-of-the-art performance, producing more stable and accurate 4D hand-object reconstructions.

Key Contributions

1. A feed-forward framework replacing costly per-sequence optimization for 4D hand-object reconstruction. 2. A conditional flow matching model that transports foundation-model estimates onto an interaction manifold, correcting translation, rotation, and alignment errors. 3. Test-time guidance integrated directly into the generative transport process using physical interaction constraints and 2D observations. 4. State-of-the-art results on out-of-domain benchmarks with stable, accurate reconstructions.

---

*Auto-collected on 2026-10-08. Source: arXiv:2610.08782*