Para-Pipe: Exploiting Hierarchical Operator Parallelism for Edge ML Inference on Heterogeneous SoCs

Forum topic · 小凯 · 2026-09-07

Summary

Para-Pipe is a hierarchical mapping framework for optimizing deep learning inference on heterogeneous System-on-Chips (SoCs), introduced in arXiv paper 2509.04277 by Yujie Zhang, Huiying Lan, and Ehsan Aghapour. While traditional pipelining improves throughput across on-chip processing units, it struggles with the latency demands of modern neural networks, and prioritizing either pipelining or operator parallelism typically trades off one metric against the other. Para-Pipe integrates intra-stage and inter-stage operator parallelism within a pipelined architecture, selectively tuning parallelism levels inside and across pipeline stages to balance throughput and latency. This approach reduces inter-processor communication overhead and improves energy efficiency. Evaluations on an Amlogic SoC with ARM big.LITTLE CPUs and GPU, and a Black Sesame Technologies SoC with a deep learning accelerator and two DSPs, show Para-Pipe produces multiple Pareto-optimal configurations. Its throughput-optimized configuration on the Amlogic SoC improves energy efficiency by 11.0% over pure pipelining and 23.3% over non-pipelined parallel execution.

Paper Overview

Field: ML Authors: Yujie Zhang, Huiying Lan, Ehsan Aghapour Published: 2026-09-06 arXiv: 2509.04277

Abstract

As edge-based deep learning applications become more complex, optimizing performance on heterogeneous System-on-Chips (SoCs) presents unique challenges. Traditional pipelining techniques distributing the computation across different on-chip processing units, while effective for throughput, do not address the latency demands posed by modern neural networks with complex interdependencies and extensive operator parallelism. There is a potential in leveraging operator parallelism to enable concurrent execution across multiple processing units, thereby reducing inference latency. However, prioritizing pipelining or parallel execution often necessitates a compromise, where optimizing one performance metric adversely impacts the other.

This paper introduces Para-Pipe, a hierarchical mapping framework that integrates intra-stage and inter-stage operator parallelism within a pipelined architecture. Para-Pipe trades off throughput and latency by selectively fine-tuning parallelism levels within and across pipeline stages. This strategy can significantly reduce inter-processor communication overhead and substantially improve energy efficiency.

Key Results

Evaluated on an Amlogic SoC with ARM big.LITTLE CPU and GPU, and a Black Sesame Technologies SoC with a deep learning accelerator and two DSPs.
Para-Pipe generates multiple Pareto-optimal configurations balancing throughput and latency.
The throughput-optimized configuration on the Amlogic SoC improves energy efficiency by an average of 11.0% over pure pipelining and 23.3% over non-pipelined parallel execution.

*Auto-collected on 2026-09-07.*