Actor-Critic without Actor (ACA): A Framework Analysis

Forum topic · ✨步子哥 · 2025-11-13

Summary

Actor-Critic without Actor (ACA) is a reinforcement learning framework that removes the explicit Actor network entirely, instead generating actions directly from the gradient field of a noise-level Critic. This article analyzes how ACA works: the Critic is trained to respond to noise perturbations in action space, producing a gradient field that guides action generation through iterative optimization. By eliminating the separate policy network, ACA reduces model complexity and computation, and fundamentally solves the policy lag problem, since actions are always aligned with the Critic's latest value estimates. Experiments on standard online RL benchmarks (MuJoCo control tasks) show ACA achieves faster learning curves and competitive final performance versus A2C/A3C and diffusion-based Actor-Critic variants, while retaining the ability to express diverse, multimodal behaviors without generative models. The analysis also discusses limitations: higher demands on the Critic, potential exploration deficits from greedy gradient-based action generation, unclear applicability to discrete action spaces, and the need for stronger theoretical guarantees. Overall, ACA demonstrates that actor-free policy generation via Critic gradients is a viable, efficient alternative in continuous control.

Introduction

Actor-Critic (AC) methods combine policy gradients with value functions, using an Actor network to generate actions and a Critic network to evaluate them. While effective for continuous action and high-dimensional state spaces, this architecture carries high model complexity and computational overhead. Recent attempts to integrate diffusion models into AC frameworks capture richer multimodal policies but further increase complexity and training difficulty.

The Actor-Critic without Actor (ACA) framework offers a new approach: it removes the explicit Actor network and generates actions directly from the gradient field of a noise-level critic. This design significantly reduces model parameters and training complexity while keeping action generation tightly aligned with the Critic's latest value estimates, thereby eliminating the policy lag problem. Experiments on standard online RL benchmarks show faster learning curves and competitive performance while retaining the ability to capture diverse, multimodal behaviors.

Key points

Traditional AC limitations: Training two networks increases parameters and compute; the Actor's updates lag behind the Critic's value estimates (policy lag); diffusion-based variants add further complexity.
Core idea: ACA drops the Actor network. The Critic is trained as a *noise-level critic* that responds to noise perturbations in action space, effectively learning a gradient field over actions: for a given state, it indicates which direction in action space increases the value estimate.
Action generation: Starting from an initial action (random or previous action), ACA performs gradient ascent along the Critic's gradient field to iteratively approach a high-value action — a policy search guided in real time by the Critic.
Eliminating policy lag: Since actions come directly from the Critic's current output, there is no separate policy network to fall behind. Policy and value updates are fully synchronized, improving efficiency and stability.

Experimental results

Benchmarks include continuous control tasks (MuJoCo: Walker2d, Hopper, HalfCheetah) compared against A2C, A3C, and diffusion-based Actor-Critic variants.
Faster convergence: ACA reaches high reward levels with fewer environment interactions due to the reduced update latency.
Competitive final performance: On most tasks, ACA matches or slightly exceeds baseline final rewards despite its simpler architecture.
Multimodal behavior: Through gradient-field-based policy search, ACA produces diverse behavior patterns (e.g., different gaits or paths), comparable to diffusion-model approaches without needing generative components.
Ablations: Variants without the noise-level critic (reverting to standard AC) or without gradient-field iteration perform significantly worse, confirming both mechanisms are essential.

Advantages

1. Simplicity and efficiency: Roughly half the parameters; lower overfitting risk and better suitability for resource-constrained settings. 2. No policy lag: Synchronized policy/value updates yield faster, more stable learning and less sample waste. 3. Expressiveness retained: Multimodal policies emerge from gradient search without generative models. 4. Simpler training: One network to tune, avoiding two-time-scale update tricks and actor-critic desynchronization issues.

Limitations

1. Higher demands on the Critic: It must handle both value estimation and policy generation, requiring sufficient capacity and stable training. 2. Exploration deficit: Gradient ascent is inherently greedy and may get stuck near local optima; extra exploration mechanisms (noise injection, stochastic ascent, ε-greedy) may be needed. 3. Continuous actions only (for now): Gradient fields do not directly apply to discrete action spaces; differentiable relaxations like Gumbel-Softmax introduce complexity and approximation error. 4. Theory gap: Convergence and optimality guarantees for ACA are not yet established.

Conclusion

ACA challenges the traditional Actor-Critic paradigm by showing that actor-free policy generation via a Critic's gradient field is feasible. It delivers lower complexity, no policy lag, competitive performance, and multimodal behavior on continuous-control benchmarks. Future work includes strengthening Critic capacity and training stability, designing principled exploration, extending to discrete or partially observable settings, and developing theoretical guarantees.