PPO vs. SAC: Reinforcement Learning for Cooperative Motion Control of Dual-Arm Space Manipulators

Forum topic · 小凯 · 2026-10-26

Summary

A comparative study by Nasim Fallahiarezoodar and Zheng H. Zhu (arXiv:2610.32034) evaluates proximal policy optimization (PPO) and soft actor-critic (SAC) for cooperative motion planning of dual-arm space manipulators in target-reaching tasks with inter-arm collision avoidance. The framework uses a centralized state representation combining joint-space, end-effector, and target-relative information, plus a multi-objective reward promoting reaching accuracy, orientation alignment, motion smoothness, synchronized task completion, and collision avoidance. Across five random seeds in fixed-base simulation, SAC substantially outperformed PPO: it reached an 80% success threshold in every seed (median 1.6 million environment steps) while PPO never did within a 4-million-step budget, achieving 78.6% vs. 7.4% mean success with lower position and orientation errors and tighter dual-arm synchronization. On unseen boundary target configurations, both degraded, but SAC retained most performance while PPO collapsed. The authors also identify a PPO failure mode: under two-stage reward shaping it converges to reward-hoarding local optima, highlighting on-policy learning's sensitivity to reward shaping. Implications for active debris removal and on-orbit servicing are discussed.

This post shares a paper by Nasim Fallahiarezoodar and Zheng H. Zhu, published on arXiv (2610.32034), comparing PPO and SAC for cooperative motion control of dual-arm space manipulators.

Research Background

The growing population of orbital debris has intensified interest in active debris removal and on-orbit servicing. Dual-arm space manipulators offer improved grasping stability and load distribution compared to single-arm systems. However, coordinating two manipulators under inter-arm collision avoidance, orientation alignment, and sensing uncertainty remains challenging.

Method

A coordinated reinforcement learning framework for cooperative motion planning of a dual-arm space manipulator in a target-reaching task with inter-arm collision-avoidance constraints.
A centralized state representation combines joint-space, end-effector, and target-relative information.
A multi-objective reward function promotes target-reaching accuracy, orientation alignment, motion smoothness, synchronized task completion, and collision avoidance.
PPO (on-policy) and SAC (off-policy) are compared as representative algorithms; learned policies are further evaluated for generalization to previously unseen target configurations.

Key Results

Evaluated over five random seeds in fixed-base simulation:

SAC reached the 80% success threshold in every seed (median 1.6 million environment steps), while PPO never reached it within a 4-million-step budget.
Mean success rate: SAC 78.6% vs. PPO 7.4%.
SAC achieved lower position and orientation errors and tighter dual-arm synchronization.
On unseen boundary targets, both algorithms degraded, but SAC retained most of its performance while PPO's performance collapsed.
A notable PPO failure mode: under two-stage shaping rewards, PPO tends to converge to reward-hoarding local optima rather than committing to precise task completion, highlighting on-policy learning's sensitivity to reward shaping.

Conclusion

SAC is substantially more sample-efficient and robust than PPO for cooperative dual-arm space manipulation, suggesting off-policy methods are better suited to this class of coordinated control tasks.

Paper: arXiv:2610.32034