Para-Pipe: Exploiting Hierarchical Operator Parallelism for Efficient DNN Inference on Heterogeneous Edge SoCs

Forum topic · 小凯 · 2026-09-07

Summary

Para-Pipe is a hierarchical mapping framework proposed to optimize deep learning inference on heterogeneous edge System-on-Chips (SoCs). Traditional pipelining across on-chip processing units improves throughput but struggles to meet the latency requirements of modern neural networks with complex dependencies and heavy operator parallelism, while prioritizing parallel execution often trades off throughput against latency. Para-Pipe integrates intra-stage and inter-stage operator parallelism within a pipelined architecture, selectively tuning parallelism levels inside and across pipeline stages to balance throughput and latency. This reduces inter-processor communication overhead and substantially improves energy efficiency. Evaluation on an Amlogic SoC with ARM big.LITTLE CPUs and a GPU, and on a Black Sesame Technologies SoC with a deep learning accelerator and two DSPs, shows that Para-Pipe produces multiple Pareto-optimal configurations. Its throughput-optimized configuration on the Amlogic SoC improves energy efficiency by 11.0% on average over pure pipelining and 23.3% over non-pipelined parallel execution. Paper: arXiv:2509.04277.

Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computations on Edge SoCs

Field: Machine Learning / Edge Computing
Authors: Yujie Zhang, Huiying Lan, Ehsan Aghapour
Published: 2026-09-06
arXiv: 2509.04277

Key points

As edge deep learning applications grow more complex, optimizing performance on heterogeneous SoCs becomes challenging. Traditional pipelining across on-chip processing units is effective for throughput but does not address the latency demands of modern neural networks with complex interdependencies and extensive operator parallelism.
Operator parallelism can enable concurrent execution across multiple processing units and reduce inference latency, but prioritizing either pipelining or parallel execution typically forces a compromise between throughput and latency.
Para-Pipe is a hierarchical mapping framework that integrates intra-stage and inter-stage operator parallelism within a pipelined architecture, selectively fine-tuning parallelism levels within and across pipeline stages.
This strategy significantly reduces inter-processor communication overhead and greatly improves energy efficiency.
Evaluations on an Amlogic SoC (ARM big.LITTLE CPU + GPU) and a Black Sesame Technologies SoC (deep learning accelerator + two DSPs) show Para-Pipe generates multiple Pareto-optimal configurations balancing throughput and latency.
On the Amlogic SoC, Para-Pipe's throughput-optimized configuration improves energy efficiency by 11.0% on average versus pure pipelining and 23.3% versus non-pipelined parallel execution.

Original Abstract (excerpt)

> As edge-based deep learning applications become more complex, optimizing performance on heterogeneous System-on-Chips (SoCs) presents unique challenges. Traditional pipelining techniques distributing the computation across different on-chip processing units, while effective for throughput, do not address the latency demands posed by modern neural networks with complex interdependencies and extensive operator parallelism. This paper introduces Para-Pipe, a hierarchical mapping framework...

*Auto-collected on 2026-09-07.*