RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture
Field: Computer Vision Authors: Tanishq Rachamalla, Srishti Kaushik, Muhammad Attique Khan, Gustau Camps-Valls, Swalpa Kumar Roy Published: 2026-11-10 arXiv: 2611.14496
Abstract
Cross-sensor remote sensing image retrieval aims to find scenes with the same content across heterogeneous sensors, such as SAR and optical or panchromatic (PAN) and multispectral (MS) imagery, without manual labels. Existing self-supervised methods either reconstruct sensor-specific pixel detail, align each image only with its co-registered twin, or rely on large transformers with 118-173M parameters (and up to 398M).
RS-JEPA is a compact joint-embedding predictive framework built on a Vision Mamba encoder shared by both sensors. Because a state-space encoder reads an image as an ordered sequence of patches, the authors propose scan-path views: two reading orders of the same image are treated as two views with identical pixels, and hidden patches in one order are predicted from visible patches in the other. To maintain prediction difficulty, Adversarial View Coupling (AVC) learns the hardest visible-patch partitions, and a neighborhood consensus loss aligns scene similarity across the two sensors.
Key Results
- Extensive experiments on SAR-optical (BEN-14K, CBRSIR-VS) and PAN-MS (DSRSID) benchmarks
- Highest F1@5 among 12 reimplemented methods: 72.92, 80.35, and 94.43 respectively
- Retrieval network of only 3.88M parameters (14.57M during training), 8-16x fewer than previous JEPA-based retrievers
- Trained on co-registered pairs, consistent with prior methods
- Highlights the potential of "reading order" as a source of self-supervised views
Code
github.com/Tanishq251/RSJEPA
--- *Auto-collected on 2026-11-12*