Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

Forum topic · 小凯 · 2026-04-18

Summary

A new arXiv paper (2504.13083) by Yiyang Jiang, Li Zhang, and Xiao-Yong Wei reframes gloss-free sign language translation (SLT) as a cross-modal reasoning task rather than direct video-to-text conversion. The authors argue that common SLT assumptions—brief signing chunks mapping directly to spoken words—fail because signers create meaning dynamically through context, space, and movement. Their reasoning-driven framework inserts an ordered sequence of latent thoughts as an explicit intermediate layer between video and generated text, gradually extracting and organizing meaning over time. Decoding follows a plan-then-ground approach: the model first decides what to say, then retrieves visual evidence, improving coherence and faithfulness. The team also releases a new large-scale gloss-free SLT dataset with stronger contextual dependence. Experiments across benchmarks consistently outperform existing gloss-free methods; code and data will be available at github.com/fletcherjiang/SignThought.

Overview

Field: Computer Vision
Authors: Yiyang Jiang, Li Zhang, Xiao-Yong Wei
Published: 2025-04-17
arXiv: 2504.13083

Abstract (translated from the Chinese summary)

Many sign language translation (SLT) systems implicitly assume that brief chunks of signing map directly to spoken-language words. This assumption breaks down because signers often create meaning on the fly using context, space, and movement. The authors revisit SLT and argue that it is primarily a cross-modal reasoning task, not merely a straightforward video-to-text conversion.

They introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit intermediate layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time.

On top of this, the framework adopts a plan-then-ground decoding method: the model first decides what it wants to say, then looks back at the video to find supporting evidence. This separation improves coherence and faithfulness.

The authors also build and release a new large-scale gloss-free SLT dataset with stronger contextual dependence and more realistic meaning construction. Experiments across multiple benchmarks consistently outperform existing gloss-free methods.

Original Abstract (excerpt)

> Many SLT systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness.

Resources

Paper: https://arxiv.org/abs/2504.13083
Code and data (upon acceptance): https://github.com/fletcherjiang/SignThought