On August 10, Alibaba DAMO Academy together with Hupan Lab uploaded the RynnValue paper to arXiv. It proposes using temporal distance as the supervision signal — replacing conventional preference labels or normalized progress — to scale a robot value foundation model to 7,000 hours and about 3 million instruction-conditioned segments. Paper: https://arxiv.org/abs/2608.09853
Why reward models are the bottleneck in robot learning
A general reward model has long been a bottleneck in robot learning: you cannot have humans write preference annotations for every subtask. RynnValue's core question: how to learn transferable value prediction from large-scale heterogeneous data without relying on intra-task anchors like preferences or progress?
Prior methods use preference or normalized progress as supervision. Both anchors have hard flaws — preferences require human labeling (expensive, slow, hard to transfer across embodiments), and normalized progress does not transfer across different task definitions (the "progress" of a pick task and a pouring task is not comparable).
RynnValue's solution: generate labels directly from timestamps.
Temporal distance as the supervision signal
Temporal distance is defined as the directed cost-to-go from an observation to the language-specified goal — i.e., "how long until task completion."
Mathematical formulation:
- Absolute temporal distance: \(v_i* = max(0, t_G - t_i)\), where \(t_G\) is the completion timestamp and \(t_i\) the observation timestamp
- Relative temporal displacement: \(Δ_i* = t_{i+1} - t_i\), positive (forward) or negative (rewind)
- Over 7,000 hours of robot trajectories
- ~3 million instruction-conditioned segments
- Online RL: +20 percentage points
- Offline RL: +18.7 percentage points
- RynnValue (no preference labels): Kendall's tau_a = 0.675
- Fully preference-supervised SOTA: 0.655
- Progress-only baseline: 0.292 (less than half)
- Total parameters: not stated in the abstract (see full paper for architecture details)
- Data scale: 7,000 hours / 3 million instruction-conditioned segments
- K=8 random observation sampling
- Time ranges: absolute [0, 512] s, relative [-256, 256] s
- 256 symlog discrete bins, two-hot encoding
- Rewind probability: 0.3
- RBM-EVAL-OOD: 0.675 Kendall's tau_a
- Real-world online RL: 72.5% (vs Robometer 52.5%)
- Real-world offline RL: 82.5% (vs Robometer 63.8%)
- Evaluation: 4 tasks, 20 trials each
- Team: DAMO Academy (Alibaba) + Hupan Lab, 15 authors with co-first and corresponding author markers
- The paper does not disclose total model parameters or detailed architecture (the abstract only covers the method)
- No public training token count / compute / specific data sources
- No public code repository (as of Aug 12, no official implementation seen on Hugging Face)
- Real-world evaluation covers only 4 tasks — generalization to long-tail tasks needs larger-scale validation
- Author affiliations are not clearly marked in the HTML version — whether Hongyin Zhang / Mingxiu Chen belong to DAMO or Hupan Lab requires checking the PDF
- https://arxiv.org/abs/2608.09853
- https://arxiv.org/html/2608.09853v1
- https://huggingface.co/papers/2608.09853
Both targets are discretized into 256 symlog bins over [0, 512] seconds and [-256, 256] seconds respectively, trained with two-hot encoding.
Key point: labels require no human annotation. Given a trajectory + task instruction + completion timestamp, the temporal distance at every observation is derivable from timestamps. Observations before the completion point are labeled with remaining time to completion; the completion point and beyond receive zero values.
This means any timestamped robot trajectory can be used directly — no human preference labels, no progress normalization. It is a fundamental shift of supervision from "human-produced" to "automatically generated from timestamps."
Three key techniques
To make temporal-distance learning reliable at scale, the paper combines three complementary techniques:
1) Random Temporal Sampling For each instruction-conditioned trajectory segment, K=8 observation points are randomly sampled at irregular intervals. This breaks the "near-arithmetic values" pattern induced by uniform sampling — the model cannot exploit a fixed sampling interval as a shortcut.
2) Temporal-order Shuffling The temporal order of sampled observations is shuffled. Half of training sequences are sampled independently without ordering (probability 0.5); the other half follow a "forward-biased temporal walk" with a rewind probability of 0.3. This prevents the model from inferring stereotyped value curves from sequence position — the rewind probability forces the model to genuinely understand time rather than extrapolate from position.
3) Value-isolation Attention Absolute/relative time queries associated with different observation points cannot attend to each other; queries within the same observation remain mutually visible, allowing feature interaction before concatenation. Additionally, context tokens are blocked from attending to time-query tokens — preventing earlier value predictions from propagating indirectly through subsequent language or visual representations.
Together these three designs suppress "value-extrapolation shortcuts": every time estimate must be based on the task instruction and its available visual context, independently, rather than referencing neighboring value predictions.
7,000 hours / 3 million instructions
RynnValue is trained on data with no preference annotations at all, scaled to:
The dataset requires no human preferences — only timestamps. This means data from different embodiments (Franka, ALOHA, xArm, etc.), tasks, and viewpoints can be merged directly.
Real-world policy improvement: 52.5% → 72.5% (online), 63.8% → 82.5% (offline)
The paper evaluates on a dual-arm Franka system (4 Intel RealSense cameras) across 4 real tasks, 20 trials each:
| Task | Online RynnValue | Online Robometer | Offline RynnValue | Offline Robometer | |------|------------------|------------------|-------------------|-------------------| | Bread Basket Placement | 45.0% | — | 100.0% | — | | Steak Serving with a Spatula | 75.0% | — | 90.0% | — | | Box-in-Drawer Placement | 70.0% | — | 90.0% | — | | Bimanual Box Transfer | 100.0% | — | 50.0% | — | | Average | 72.5% | 52.5% | 82.5% | 63.8% |
Gains:
These are unweighted averages over 4 tasks, each with 20 trials. The paper converts value predictions into dense rewards via potential-based shaping.
Benchmark: Kendall's tau_a of 0.675 on RBM-EVAL-OOD
On RBM-EVAL-OOD (out-of-distribution evaluation):
RynnValue beats fully preference-supervised SOTA without any preference labels, while more than doubling the progress-only baseline. This means temporal distance as a supervision signal is not only cheaper (no human labeling) but also more effective.
Moreover, the paper states: zero-shot transfer to unseen tasks, embodiments, and viewpoints. This is the core promise of a foundation model — swap the robot, task, or camera angle and it still works without retraining.
RynnValue at a glance
Supervision shifts from human output to timestamps
RynnValue's temporal distance is not just another clever labeling trick. It lowers the data threshold for robot value foundation models from "datasets with human preference labels" to "datasets with timestamps."
That expands the usable data pool by orders of magnitude. Any timestamped robot trajectory — whether from DROID, RT-1, Open X-Embodiment, or companies' internal collections — can directly train a value model. Supervision moves from "expensive human output" to "cheap automatic generation from timestamps."
Since August, the embodied AI mainstream has been extending from "VLA / world models / on-device inference" toward "reward signals + foundation models + data factories" — RynnValue is a flagship work in the "reward signal" sub-track. The most interesting thing to track is whether the temporal-distance idea spreads to other domains needing supervision signals: video prediction, speech synthesis, robot action generation — any timestamped multimodal data could fit the paradigm.
Limitations and unknowns
---
Sources