QF3: Fast Flow RL with Filtered Q-Gradients

Forum topic · 小凯 · 2026-10-08

Summary

QF3 (Fast Flow RL with Filtered Q-Gradients) is an online off-policy reinforcement learning algorithm for training flow-based robot policies, presented by researchers including Chung Min Kim, Brent Yi, Ken Goldberg, Pieter Abbeel, and Angjoo Kanazawa (arXiv:2610.08789). Flow policies are now a standard approach for learning robot behaviors from demonstrations, but RL remains essential for improving pre-trained policies or learning from scratch via interaction. QF3 combines flow matching with the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates within regions where the critic and prediction remain reliable, the critic gradient is applied only to action dimensions that stay near the replay action. According to the authors, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware. With a high-throughput off-policy training pipeline, it trains humanoid locomotion and action-tracking policies roughly 10x faster than the recent on-policy flow RL method FPO++. QF3 also fine-tunes pre-trained flow-based manipulation policies on ABC-Sim and Robomimic tasks, demonstrating that it can both learn policies from scratch and improve demonstration-derived policies.

Paper Overview

Field: Machine Learning / Robot Learning
Authors: Chung Min Min Kim, Brent Yi, David McAllister, Hongsuk Choi, Himanshu Gaurav Singh, Jinkun Cao, Ken Goldberg, Pieter Abbeel, Carmelo Sferrazza, Angjoo Kanazawa
Published: 2026-10-06
arXiv: 2610.08789

Summary

Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. The authors introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output.

To keep updates where the critic and this prediction are reliable, QF3 applies the critic gradient only to action dimensions that stay near the replay action.

Key Contributions

Filtered Q-gradients: critic gradients are applied only to action dimensions that remain close to the replay action, keeping updates in reliable regions.
First off-policy flow RL for humanoid locomotion: to the authors' knowledge, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware.
10x faster training: paired with a high-throughput off-policy training scheme, QF3 trains humanoid locomotion and action-tracking policies about 10x faster than the recent on-policy flow RL method FPO++.
Fine-tuning: QF3 is further applied to fine-tune pre-trained flow-based manipulation policies on ABC-Sim and Robomimic tasks.

These results show that QF3 can both learn robot policies from scratch and improve policies acquired from demonstrations.

QF3: Fast Flow RL with Filtered Q-Gradients

Summary

Paper Overview

Summary

Key Contributions

Links

Tags