Paper Overview
Field: cs.LG, math.OC Authors: Matteo Tomasetto, Nicolò Botteghi, Gabriele Bruni Published: 2026-07-21 arXiv: 2507.15484
Abstract
Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interactions with the environment to synthesize optimal control policies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces.
In this work, the authors bridge RL and traditional optimal control for dynamical systems with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentiability of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions while mitigating long-term gradient instabilities.
Through two challenging parametric navigation problems in unsteady flows, the authors show that PEARL:
- (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms;
- (ii) is sample efficient, thanks to physics-guided policy learning;
- (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems;
- (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.