Overview
Alibaba's DAMO Academy and Hupan Lab uploaded the RynnValue paper to arXiv, proposing temporal distance as a supervision signal for robotic value foundation models. The approach replaces human preference labels and normalized-progress anchors with timestamps that can be computed automatically, enabling scaling to over 7,000 hours and roughly 3 million instruction-conditioned trajectory segments.
Paper: https://arxiv.org/abs/2608.09853
Why reward modeling is the bottleneck
General-purpose reward models for robotics are a known bottleneck: it is impractical to collect human preference labels for every sub-task across embodiments. Two common anchoring strategies have drawbacks:
- Preference labels require costly, slow human annotation and are hard to transfer across embodiments.
- Normalized progress assumes comparable progress definitions across tasks that may differ fundamentally (e.g., grasping versus pouring).
- Absolute temporal distance: $v_i^* = \max(0, t_G - t_i)$, where $t_G$ is the completion deadline timestamp and $t_i$ is the current observation timestamp.
- Relative temporal displacement: $\Delta_i^* = t_{i+1} - t_i$, positive for forward motion or negative for backtracking.
- More than 7,000 hours of robot trajectories
- Approximately 3 million instruction-conditioned segments
- Online RL: +20 percentage points
- Offline RL: +18.7 percentage points
- RynnValue (no preference labels): Kendall's tau_a = 0.675
- Fully preference-supervised SOTA: 0.655
- Progress-only baseline: 0.292
- Total parameters: not stated in the abstract
- Data scale: 7,000 hours / ~3 million instruction-conditioned segments
- K = 8 random observation-point samples
- Absolute time range:
[0, 512]seconds; relative range:[-256, 256]seconds - 256 symlog discretization bins, two-hot encoding
- Rewind probability: 0.3
- RBM-EVAL-OOD: 0.675 Kendall's tau_a
- Real-world online RL: 72.5% (vs. Robometer 52.5%)
- Real-world offline RL: 82.5% (vs. Robometer 63.8%)
- Evaluation: 4 tasks × 20 trials
- Authors: 15 authors from DAMO Academy (Alibaba) + Hupan Lab
- Total parameter count and architectural details are not specified in the abstract.
- Training token count, training compute, and exact data sources are undisclosed.
- No official code repository was found on Hugging Face at the time of writing.
- Real-world evaluation covers only 4 tasks; long-tail generalization needs broader validation.
- Author affiliations between DAMO Academy and Hupan Lab are not fully clarified in the HTML version.
- https://arxiv.org/abs/2608.09853
- https://arxiv.org/html/2608.09853v1
- https://huggingface.co/papers/2608.09853
RynnValue's research question: how can transferable value prediction be learned from large-scale heterogeneous data without relying on preference or progress anchors? Its answer is to generate labels directly from timestamps.
Temporal distance as supervision
Temporal distance is defined as the directed cost-to-go from an observation to a language-specified goal — essentially "how long until the task is completed."
Both targets are discretized into 256 symlog bins using ranges of [0, 512] seconds and [-256, 256] seconds, then trained with two-hot encoding.
Crucially, no human annotation is required. Given a trajectory, an instruction, and a completion timestamp, the temporal distance for each observation can be derived directly. Observations before the completion point are labeled with remaining time; the completion point and after receive zero. Any timestamped robot trajectory can therefore be reused as training data.
Three key techniques
1. Random Temporal Sampling
For every instruction-conditioned segment, K=8 observation points are sampled with irregular time intervals. This breaks the "near-arithmetic" pattern induced by uniform sampling and prevents the model from exploiting fixed sampling intervals as a shortcut.2. Temporal-order Shuffling
The temporal order of sampled observations is shuffled. Half of training sequences are sampled independently and remain unordered (probability 0.5); the other half follow a forward-biased temporal walk with rewind probability 0.3. This prevents the model from inferring progress curves from sequence position — backtracking forces genuine temporal understanding.3. Value-isolation Attention
Value-isolation attention prevents absolute/relative time queries from different observations from attending to each other. Queries within the same observation remain visible to each other, allowing feature interaction before concatenation. Context tokens are also blocked from attending to time-query tokens, preventing prior value predictions from leaking through subsequent language or visual representations.Combined, these suppress value-extrapolation shortcuts: each time estimate must be judged independently from the task instruction and its available visual context.
Scale: 7,000 hours / 3 million instructions
RynnValue is trained without any preference labels on:
Any timestamped robot data from Franka, ALOHA, xArm, or other embodiments can be merged directly.
Real-world policy gains: 52.5% → 72.5% online, 63.8% → 82.5% offline
Evaluated on a dual-arm Franka system with four Intel RealSense cameras across four real-world tasks, 20 trials each:
| Task | Online RynnValue | Online Robometer | Offline RynnValue | Offline Robometer | |------|------------------|------------------|-------------------|-------------------| | Bread Basket Placement | 45.0% | — | 100.0% | — | | Steak Serving with a Spatula | 75.0% | — | 90.0% | — | | Box-in-Drawer Placement | 70.0% | — | 90.0% | — | | Bimanual Box Transfer | 100.0% | — | 50.0% | — | | Average | 72.5% | 52.5% | 82.5% | 63.8% |
Gains:
Values are unweighted averages across the four tasks with 20 trials per task. Value predictions are converted into dense rewards via potential-based shaping.
Benchmark: RBM-EVAL-OOD Kendall's tau_a 0.675
On the out-of-distribution evaluation benchmark RBM-EVAL-OOD:
RynnValue outperforms fully preference-supervised SOTA without any preference labels, more than doubling the progress-only baseline. The paper also emphasizes zero-shot transfer to unseen tasks, embodiments, and viewpoints.
Key data points
Significance
RynnValue shifts the data threshold for value foundation models from "datasets requiring human preference labels" to "datasets requiring only timestamps." Timestamped robot trajectories from DROID, RT-1, Open X-Embodiment, and internal corpora can all be reused. Supervision moves from an expensive human artifact to an automatic timestamp derivation.
The work fits into a broader embodied-AI thread expanding beyond VLA / world-model / on-device inference toward reward signals + foundation models + data factories, and represents a key entry in the "reward signal" sub-track. The same paradigm may extend to video prediction, speech synthesis, and robot action generation — any timestamped multimodal data.