RynnValue: Using Time-Distance Labels to Train Robot Value Models on 7,000 Hours of Data

Forum topic · 小凯 · 2026-08-14

Summary

RynnValue (arXiv 2608.09853) proposes a new approach to training value models for general robot policies by replacing expensive human preference or progress annotations with automatically generated time-distance labels. Each observation in a trajectory receives a cost-to-go label derived purely from timestamps, making supervision embodiment-agnostic and scalable to millions of hours. To prevent shortcut learning on raw time, the authors apply random temporal sampling, sequence shuffling, and value-isolation attention. Trained on 7,000+ hours (about 3 million instruction-conditioned clips) without human labeling, the model reaches Kendall's tau_a of 0.675 on RBM-EVAL-OOD, beating the preference-supervised state of the art (0.655) and improving over progress-only baselines (0.292) by 2.3x. Real-world policy success rates rise from 52.5% to 72.5% online and from 63.8% to 82.5% offline, with zero-shot transfer to unseen tasks, embodiments, and viewpoints.

Overview

RynnValue (arXiv 2608.09853, 23 pages, 5 figures, submitted 2026-08-10) reframes robot value modeling: instead of costly human preference pairs or hand-defined progress stages, it uses the timestamp on every trajectory segment as the supervision signal. The directed cost-to-go from each observation to a language-specified goal becomes a regression label, generated automatically and consistent across embodiments.

Why Time-Distance Works as Supervision

Free labels: every clip already carries timestamps, so cost-to-go is computed in O(1) with no annotators
Embodiment-agnostic: time-distance carries no morphology-specific features
Zero-shot transfer: because supervision is decoupled from embodiment, the learned value function generalizes to unseen tasks, robots, and camera viewpoints

To stop the model from collapsing onto a pure "is it early or late?" shortcut, three training tricks are stacked:

1. Random temporal sampling across trajectories 2. Within-task segment shuffling 3. Value-isolation attention that separates failure/regress sensitivity from time-point prediction

Hard Numbers on RBM-EVAL-OOD

Training data: 7,000+ hours, ~3M instruction-conditioned clips, zero preference labels, zero progress labels
Kendall's tau_a = 0.675, exceeding the preference-supervised SOTA of 0.655
2.3x improvement over the progress-only baseline (0.292)
Real-world success rate:
Online: 52.5% → 72.5% (+20.0 pp)
Offline: 63.8% → 82.5% (+18.7 pp)
Zero-shot transfer demonstrated to unseen tasks, unseen embodiments, and new viewpoints

Preference vs Time-Distance: A Data Paradigm Shift

| Dimension | Preference labels | Time-distance | |---|---|---| | Label source | Humans compare two trajectories | Timestamps on existing data | | Annotation cost | O(n²) pairwise | O(1) automatic | | Embodiment transfer | Weak (preference is embodiment-specific) | Strong (cross-embodiment) | | Data ceiling | ~10k hours (human-limited) | ~10M hours (automated) | | Failure sensitivity | Strong | Moderate, requires architectural fix |

The headline inversion is real: an automatically labeled model now outperforms the human-preference SOTA on ranking quality, suggesting the bottleneck in embodied value modeling is annotation cost, not architecture.

Dense Reward as a Universal Interface

The value function is converted into a potential-based shaped reward, which preserves policy optimality under standard RL theory. This makes the output plug-and-play:

Any RL algorithm (PPO, SAC, IQL) consumes it as a standard dense reward
It can serve as a critic alongside VLA actors such as π0, OpenVLA, or Helix
Sim-to-real is straightforward because no embodiment-specific recalibration is required

The design parallels process reward models used as verifiers in reasoning systems like OpenAI o1: generalizing the reward signal is positioned as a key step toward industrial deployment of embodied intelligence.

Limitations

1. Timestamp reliability: cross-vendor robot fleets introduce clock drift; the paper assumes a single-device cluster 2. Well-defined task structure: when goals are ambiguous ("make the user happy"), time-distance collapses into predicting segment length 3. Extreme embodiment OOD: zero-shot transfer is shown across robots, but leg → wheel → aerial morphology gaps are not fully validated

What to Watch Next

Whether the model weights are released alongside the paper
Cross-lab replication by NVIDIA GR00T, Google DeepMind RT-2, and Physical Intelligence π0 teams on the same time-distance recipe
Measured gains when RynnValue is wired in as the critic inside OpenVLA, π0, or Helix

---

Key data points: 7,000+ hours, ~3M clips, tau_a 0.675 vs SOTA 0.655 vs progress 0.292, online 52.5%→72.5%, offline 63.8%→82.5%, 23 pages / 5 figures, arXiv 2608.09853 (cs.RO, 2026-08-11)

Sources: arXiv, HuggingFace Daily Papers (2026-08-10)