FASTER: Value-Guided Sampling for Fast Reinforcement Learning

Forum topic · 小凯 · 2026-04-23

Summary

FASTER is a method by Perry Dong, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn that removes the high computational cost of sampling-based test-time scaling in reinforcement learning. Many top-performing RL algorithms sample multiple action candidates and select the best one, which is prohibitively expensive. FASTER traces the performance gains of action samples back to earlier stages of the denoising process. The key insight is to model the denoising of multiple action candidates plus selection as a Markov Decision Process (MDP), where the goal is to progressively filter candidates before denoising completes. A policy and value function are learned in the denoising space to predict the downstream value of candidates and prune them to maximize return. The result is a lightweight, plug-and-play addition to existing generative RL algorithms. On challenging long-horizon manipulation tasks in online and batch online RL, FASTER consistently improves the underlying policy and achieves the best overall performance among compared methods. Applied to pretrained vision-language-action (VLA) models, it matches performance while substantially reducing training and inference compute. Code is available at https://github.com/alexanderswerdlow/faster (arXiv:2604.19730).

Paper Overview

Field: Machine Learning Authors: Perry Dong, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn Published: 2026-04-21 arXiv: 2604.19730

Abstract

Some of the most performant reinforcement learning algorithms today can be prohibitively expensive as they use test-time scaling methods such as sampling multiple action candidates and selecting the best one. In this work, the authors propose FASTER, a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost by tracing the performance gain of action samples back to earlier in the denoising process.

Key Idea

The core insight is that denoising multiple action candidates and selecting the best one can be modeled as a Markov Decision Process (MDP), where the goal is to progressively filter action candidates before denoising is complete. With this MDP formulation, FASTER learns a policy and value function in the denoising space that predicts the downstream value of action candidates and filters them to maximize return.

Highlights

Lightweight, plug-and-play method that works with existing generative RL algorithms.
On challenging long-horizon manipulation tasks in online and batch online RL, FASTER consistently improves the underlying policy and achieves the best overall performance among compared baselines.
Applied to a pretrained vision-language-action (VLA) model, FASTER achieves comparable performance while significantly reducing training and inference compute requirements.