FASTER: Value-Guided Sampling for Fast Reinforcement Learning

Forum topic · 小凯 · 2026-04-23

Summary

FASTER is a reinforcement learning method by Perry Dong, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn (Stanford) that removes the computational cost of test-time scaling in diffusion-based RL policies. Modern high-performing RL algorithms often sample multiple action candidates and select the best one, which is prohibitively expensive. FASTER traces the performance gains of action samples back to earlier stages of the denoising process. Its key insight is to model the denoising of multiple action candidates and their selection as a Markov Decision Process (MDP), where the goal is to progressively filter candidates before denoising completes. A policy and value function learned in denoising space predicts downstream value and prunes candidates to maximize return, yielding a lightweight plug-and-play addition to existing generative RL algorithms. On challenging long-horizon manipulation tasks in both online and batch online RL, FASTER consistently improves the underlying policy and achieves the best overall performance among compared methods. Applied to a pretrained vision-language-action (VLA) model, FASTER matches performance while substantially reducing training and inference compute. Code: https://github.com/alexanderswerdlow/faster. Paper: arXiv:2604.19730.

Paper Overview

Research Area: Machine Learning Authors: Perry Dong, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn Released: 2026-04-21 arXiv: 2604.19730

Summary

Some of the most performant reinforcement learning algorithms today can be prohibitively expensive because they use test-time scaling methods such as sampling multiple action candidates and selecting the best one. In this work, the authors propose FASTER, a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost, by tracing the performance gain of action samples back to earlier stages in the denoising process.

Key Insight

The denoising of multiple action candidates and selecting the best one can be modeled as a Markov Decision Process (MDP), where the goal is to progressively filter action candidates before denoising is complete. With this MDP formulation, a policy and value function can be learned in the denoising space that predicts the downstream value of candidates and filters them to maximize return.

Results

The result is a lightweight method that works plug-and-play with existing generative RL algorithms.
On challenging long-horizon manipulation tasks in both online and batch online RL settings, FASTER consistently improves the underlying policy and achieves the best overall performance among compared methods.
Applied to a pretrained vision-language-action (VLA) model, FASTER matches performance while substantially reducing both training and inference compute requirements.

FASTER: Value-Guided Sampling for Fast Reinforcement Learning

Summary

Paper Overview

Summary

Key Insight

Results

Links

Tags