AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation

Forum topic · 小凯 · 2026-03-28

Summary

AnyHand is a large-scale synthetic dataset for 3D hand pose estimation from RGB-only and RGB-D inputs, containing 2.5M single-hand and 4.1M hand-object interaction RGB-D images with rich geometric annotations. Unlike prior real-world and synthetic datasets, it provides occlusions, arm details, and aligned depth at scale. Adding AnyHand to existing training sets significantly improves performance on FreiHAND and HO-3D benchmarks without architectural changes, and models trained on AnyHand generalize better to out-of-domain HO-Cap data without fine-tuning. The authors also contribute a lightweight depth-fusion module easily integrated into existing RGB models; when trained with AnyHand, the resulting RGB-D model achieves state-of-the-art results on HO-3D, demonstrating the value of depth integration and synthetic data. Paper: arXiv 2603.25726, by Chen Si, Yulin Liu, Bo Ai, Jianwen Xie, Rolandos Alexandros Potamias, Chuanxia Zheng, and Hao Su.

Paper Overview

Field: Computer Vision Authors: Chen Si, Yulin Liu, Bo Ai, Jianwen Xie, Rolandos Alexandros Potamias, Chuanxia Zheng, Hao Su Released: 2026-03-26 arXiv: 2603.25726

Key Contributions

AnyHand dataset: a large-scale synthetic dataset for 3D hand pose estimation supporting both RGB-only and RGB-D inputs.
Scale and coverage: 2.5M single-hand and 4.1M hand-object interaction RGB-D images with rich geometric annotations, including occlusions, arm details, and aligned depth — features rarely provided together at scale by prior synthetic datasets.
RGB-only results: extending existing baselines' training sets with AnyHand yields significant gains on FreiHAND and HO-3D benchmarks, with unchanged architecture and training scheme.
Generalization: models trained with AnyHand show stronger generalization to the out-of-domain HO-Cap dataset without any fine-tuning.
Depth fusion: a lightweight depth-fusion module that can be easily integrated into existing RGB models. Trained with AnyHand, the resulting RGB-D model achieves superior performance on the HO-3D benchmark.

Abstract (original)

We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation from both RGB-only and RGB-D inputs. While recent works with foundation approaches have shown that an increase in the quantity and diversity of training data can markedly improve performance and robustness in hand pose estimation, existing real-world-collected datasets on this task are limited in coverage, and prior synthetic datasets rarely provide occlusions, arm details, and aligned depth together at scale. To address this bottleneck, our AnyHand contains 2.5M single-hand and 4.1M hand-object interaction RGB-D images, with rich geometric annotations. In the RGB-only setting, we show that extending the original training sets of existing baselines with AnyHand yields sig...

--- *Auto-collected on 2026-03-28*