Para-Pipe: Exploiting Hierarchical Operator Parallelism for Edge Deep Learning on Heterogeneous SoCs

Forum topic · 小凯 · 2026-09-07

Summary

Para-Pipe is a hierarchical mapping framework that integrates intra-stage and inter-stage operator parallelism within a pipelined execution architecture for deep learning inference on heterogeneous System-on-Chips (SoCs). Traditional pipelining distributes computation across on-chip processing units and improves throughput, but struggles to meet latency demands of modern neural networks with complex dependencies. Naively prioritizing pipelining or parallel execution involves a trade-off between throughput and latency. Para-Pipe selectively tunes parallelism levels within and across pipeline stages, balancing these metrics while reducing inter-processor communication overhead and improving energy efficiency. Evaluations on an Amlogic SoC with ARM big.LITTLE CPUs and a GPU, and a Black Sesame Technologies SoC with a deep learning accelerator and two DSPs, show that Para-Pipe generates multiple Pareto-optimal configurations. On the Amlogic SoC, its throughput-optimized configuration improves energy efficiency by 11.0% on average over pure pipelining and by 23.3% over non-pipelined parallel execution. Paper: arXiv:2509.04277.

论文概要

研究领域: ML 作者: Yujie Zhang, Huiying Lan, Ehsan Aghapour 发布时间: 2026-09-06 arXiv: 2509.04277

English Translation

As edge-based deep learning applications become more complex, optimizing performance on heterogeneous System-on-Chips (SoCs) presents unique challenges. Traditional pipelining techniques distribute the computation across different on-chip processing units. While effective for throughput, they do not address the latency demands posed by modern neural networks with complex interdependencies and extensive operator parallelism. There is potential in leveraging operator parallelism to enable concurrent execution across multiple processing units, thereby reducing inference latency. However, prioritizing pipelining or parallel execution often necessitates a compromise, where optimizing one performance metric adversely impacts the other.

This paper introduces Para-Pipe, a hierarchical mapping framework that integrates intra-stage and inter-stage operator parallelism within a pipelined architecture. Para-Pipe trades off throughput and latency by selectively tuning parallelism levels within and across pipeline stages. This strategy significantly reduces inter-processor communication overhead and greatly improves energy efficiency.

Key Results

Evaluated on an Amlogic SoC with ARM big.LITTLE CPUs and a GPU, and a Black Sesame Technologies SoC with a deep learning accelerator and two DSPs.
Para-Pipe generates multiple Pareto-optimal configurations balancing throughput and latency.
On the Amlogic SoC, the throughput-optimized configuration improves energy efficiency by 11.0% on average over pure pipelining and 23.3% over non-pipelined parallel execution.

---

*Auto-collected on 2026-09-07*