Actor-Critic without Actor (ACA): A Reinforcement Learning Framework Analysis

Introduction

Actor-Critic (AC) methods combine policy gradients with value functions and are a cornerstone of modern reinforcement learning. In traditional AC, an actor network generates actions from the current state, while a critic network evaluates the policy's value to guide improvement. This architecture handles continuous action and high-dimensional state spaces well, but adds model complexity and computation cost. Recent work has even integrated diffusion models into AC frameworks to capture multimodal policy distributions, at the cost of further complexity.

The Actor-Critic without Actor (ACA) framework takes a different route: it removes the explicit actor network and instead generates actions directly from the gradient field of a noise-level critic. This reduces model size and training complexity, aligns action generation with the critic's latest value estimates, and eliminates the policy lag problem. Experiments show faster learning curves, competitive performance, and preserved capability for diverse, multimodal behaviors.

Methodology and Technical Details

Limitations of Traditional Actor-Critic

• AC methods balance the high variance of policy gradients with the bias of value-based methods. The actor outputs an action distribution; the critic estimates \(V(s)\) or \(Q(s,a)\) to compute the advantage \(A(s,a)=Q(s,a)-V(s)\).
• Model complexity: two networks must be trained simultaneously, increasing parameters and compute.
• Policy lag: the actor's updates depend on the critic's estimates, which are themselves noisy and lagging, so policy improvements can be delayed or insufficient.
• Generative additions like diffusion models improve expressiveness but further increase training difficulty.

Core Idea of ACA

ACA's central idea: eliminate the explicit actor network and use the critic's gradient field to generate actions directly. The critic is trained to respond to noise-level perturbations in the action space. Given state \(s\) and a perturbed action \(a' = a + \epsilon\), the noise-level critic outputs evaluations such as \(Q(s, a')\) or gradient information, effectively learning a gradient field over the action space indicating which direction improves value.

Action generation works by starting from an initial action and performing gradient ascent along the critic's gradient field, iteratively approaching a high-value action — a form of policy search guided in real time by the critic. Benefits:

1. Actor-free design roughly halves parameters and reduces compute. 2. Tight alignment between action generation and the critic's latest estimates, so policy lag is eliminated — improvements in value estimates are immediately reflected in actions.

Training the Noise-Level Critic

Unlike conventional critics trained via temporal-difference learning alone, the noise-level critic must learn responses to action perturbations, e.g., by targeting \(\nabla_a Q(s,a)\) via automatic differentiation, or through perturbation-contrastive learning to distinguish high- and low-value perturbations. The goal is a differentiable policy generator that outputs high-quality actions directly from the state.

Resolving Policy Lag

In traditional AC, alternating updates mean the actor's improvement trails the critic's learning, hurting convergence and stability. In ACA there is no separate policy network to wait for: when the critic updates its gradient field, action generation immediately reflects it, achieving synchronous policy-value updates and more stable, efficient learning.

Experiments and Results

Setup

ACA was implemented as a critic-only model trained as a noise-level critic, with actions generated by iterative gradient optimization. Benchmarks covered continuous control tasks (e.g., MuJoCo environments like Walker2d, Hopper, HalfCheetah) and comparisons against A2C/A3C, diffusion-based actor-critic variants, and other strong baselines. Metrics: learning curves, final performance, and behavioral diversity. Multiple runs per task assessed stability.

Performance

• Faster convergence: ACA typically reaches high reward levels sooner than traditional AC methods, thanks to removing actor-update latency.
• Competitive final performance: on most tasks ACA matches or slightly exceeds A2C/A3C; against diffusion-based variants it stays competitive while training more efficiently.
• Multimodal behavior: because gradient-field action generation performs policy search over continuous action space, ACA produces diverse behavior patterns (e.g., different gaits or action paths), comparable to diffusion-model AC but without extra generative components.

Ablation Studies

Variants without the noise-level critic (degenerating to standard AC) or without gradient-field iteration performed clearly worse — slower learning and lower final rewards — confirming that both the actor-free design and gradient-based action generation are essential to ACA's advantages.

Advantages and Limitations

Advantages

1. Simplicity and efficiency: ~half the parameters; lower compute, memory, and overfitting risk. 2. No policy lag: synchronous policy-value updates improve sample efficiency, real-time responsiveness, and stability. 3. Preserved expressiveness: captures complex and multimodal policies without generative models. 4. Simpler training: one network means no two-time-scale tricks or synchronization tuning, and avoids AC oscillation issues.

Limitations

1. Higher demands on the critic: it must handle both value estimation and policy generation, requiring sufficient capacity and stable training. 2. Insufficient exploration: gradient ascent is inherently greedy and may stagnate near local optima; extra exploration mechanisms (noise injection, stochastic ascent, ε-greedy) may be needed. 3. Limited to continuous actions: gradient fields do not directly apply to discrete action spaces; differentiable relaxations (e.g., Gumbel-Softmax) add complexity and approximation error. 4. Underdeveloped theory: convergence and stability guarantees for ACA have yet to be established.

Conclusion

ACA demonstrates the feasibility of an actor-free reinforcement learning paradigm: a noise-level critic's gradient field can serve as a lightweight, expressive policy generator that eliminates policy lag, speeds learning, and retains multimodal behavior — at competitive benchmark performance. Future directions include strengthening critic capacity and training stability, designing principled exploration mechanisms, extending to discrete or partially observable settings, and developing formal convergence theory. ACA challenges the traditional actor-critic paradigm and offers a promising template for simpler, more efficient RL agents.