RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation

Forum topic · 小凯 · 2026-06-27

Summary

RayPE is a positional-encoding extension for video diffusion transformers proposed by Minghao Yin, Jiahao Lu, and Wenbo Hu (arXiv:2606.27345). Modern video diffusion transformers use RoPE to position tokens along the (u,v,t) axes, which describes the camera's sampling grid but is unrelated to the 3D structure of the scene. The key insight behind RayPE is that the geometric relation between two camera rays is captured by the Plucker reciprocal product, a bilinear form identical in algebraic structure to the dot product used in Transformer attention. Building on this analogy, RayPE injects ray geometry directly into the attention mechanism, making positional encoding 3D-aware. This enables video generation models to better reason about camera viewpoints and scene geometry, improving camera controllability and 3D consistency. The work falls in the computer vision (CV) field and was published on arXiv on 2026-06-27.

Paper Overview

Field: Computer Vision (CV)
Authors: Minghao Yin, Jiahao Lu, Wenbo Hu
Published: 2026-06-27
arXiv: 2606.27345

Key Idea

Modern video diffusion transformers position their tokens through RoPE on the (u,v,t) axes — a description of the camera's sampling grid that says nothing about the 3D structure of the scene. The authors observe that the geometric relation between two camera rays is captured by the Plucker reciprocal product, which is bilinear in the two rays — the same algebraic form as the dot product in Transformer attention.

Contribution

Building on this analogy, the paper proposes RayPE, a positional-encoding extension that injects ray geometry directly into the attention mechanism, enabling 3D-aware positional encoding for video generation.

Original Abstract (excerpt)

> Modern video diffusion transformers position their tokens through RoPE on the (u,v,t) axes -- a description of the camera's sampling grid that says nothing about the 3D structure of the scene. We observe that the geometric relation between two camera rays is captured by the Plucker reciprocal product, which is bilinear in the two rays -- the same algebraic form as the dot product in Transformer attention. Building on this analogy, we propose RayPE, a positional-encoding extension that injects pe...

*Auto-collected on 2026-06-27 from zhichai.net.*