SignRR: Retrieve and Refine Real Motion for Sign Language Production

Forum topic · 小凯 · 2026-09-01

Summary

SignRR is a retrieve-and-refine framework for sign language production (SLP) that generates continuous signing motion from spoken language text. Prior approaches follow two paradigms: generative models synthesize motion from a learned prior or noise, making rare hand configurations and signer-specific articulation hard to preserve, while retrieval-based methods reuse real motion segments but introduce rhythm and style inconsistencies when concatenating clips from different signers and co-articulation contexts. SignRR combines both paradigms: it initializes motion from a dictionary of real sign language clips, then refines the full sequence using a part-aware residual VQ-VAE, where residual quantization preserves fine hand articulation and temporal length differences are handled in latent space. Experiments on PHOENIX14T and CSL-Daily show that SignRR achieves state-of-the-art back-translation performance while maintaining competitive pose quality.

Paper Overview

Field: Computer Vision (CV)
Authors: Fidel Omar Tito Cruz, Angie Sanchez Marquina, Summy Farfan, Gissella Bejarano
Published: 2026-08-28
arXiv: 2608.28568

Abstract

Sign language production (SLP) aims to generate continuous signing motion from spoken language, often through gloss-to-pose generation. Prior work mainly follows two paradigms:

Generative models synthesize motion from a learned prior or from noise, without reference to an observed signing instance, making rare hand configurations and signer-specific articulation difficult to preserve.
Retrieval-based methods reuse real, well-articulated motion segments, but concatenating segments from different signers and co-articulation contexts can introduce rhythm and style inconsistencies across the full sequence, not only at segment boundaries.

These limitations suggest a complementary solution: use retrieval to provide realistic articulation, and use learned refinement to impose the global coherence that retrieval alone lacks.

SignRR Framework

SignRR adopts a retrieve-and-refine paradigm: instead of generating motion from scratch, it starts from real retrieved motion and refines it into a globally coherent sign language sequence.

1. Retrieval initialization: motion is initialized from a dictionary of real sign language clips. 2. Refinement: the complete sequence is refined with a part-aware residual VQ-VAE, where residual quantization preserves fine-grained hand articulation and temporal length differences between retrieved segments are handled in latent space.

Results

Experiments on PHOENIX14T and CSL-Daily demonstrate that SignRR achieves state-of-the-art back-translation performance while maintaining competitive pose quality.

--- *Auto-collected on 2026-09-01*