Q-based Variational Inverse Reinforcement Learning (QVIRL): Bayesian Reward Recovery from Expert Demonstrations

Forum topic · 小凯 · 2026-08-19

Summary

QVIRL is a novel Bayesian inverse reinforcement learning method introduced by Ondrej Bajgar, Peter Tisnikar, Alessandro Abate and colleagues (arXiv:2608.16888). Safe and beneficial AI requires systems that learn and act according to human preferences, but hand-specifying these preferences is often infeasible; IRL addresses this by inferring reward functions from expert behaviour. QVIRL recovers a posterior distribution over rewards from expert demonstrations by primarily learning a variational distribution over optimal Q-values. Unlike previous approaches, it combines scalability with uncertainty quantification, which matters for safety-critical applications and active learning. The authors demonstrate strong apprenticeship-learning performance across gridworlds, Lunar Lander, highway environments, and two ATARI games, using both static expert datasets and active learning. Notably, it is the first Bayesian IRL method shown to train from raw pixel observations.

Paper Overview

Field: Machine Learning
Authors: Ondrej Bajgar, Peter Tisnikar, Alessandro Abate et al. (5 authors)
Posted: 2026-08-17
arXiv: 2608.16888

Abstract

The development of safe and beneficial AI requires that systems can learn and act in accordance with human preferences. However, explicitly specifying these preferences by hand is often infeasible. Inverse reinforcement learning (IRL) addresses this challenge by inferring preferences, represented as reward functions, from expert behaviour.

We introduce Q-based Variational IRL (QVIRL), a novel Bayesian IRL method that recovers a posterior distribution over rewards from expert demonstrations via primarily learning a variational distribution over optimal Q-values. Unlike previous approaches, QVIRL combines scalability with uncertainty quantification, important for safety-critical applications as well as active learning.

Key Points

Learns a variational distribution over optimal Q-values as the primary route to recovering a posterior over reward functions.
Combines scalability with uncertainty quantification, valuable for safety-critical applications and for active learning.
Demonstrated on gridworlds, Lunar Lander, highway environments, and two ATARI games in apprenticeship-learning tasks.
Works with both static expert datasets and active learning.
First Bayesian IRL method shown to train from raw pixel observations.

---

*Auto-collected on 2026-08-19.*