Strait: Perceiving Priority and Interference in ML Inference Serving

Forum topic · 小凯 · 2026-05-02

Summary

Strait is an ML inference serving system designed to improve deadline satisfaction for dual-priority inference traffic under high GPU utilization. Addressing the limited prioritization support and inaccurate latency estimation of existing serving systems in on-premises deployments, Strait models potential contention during data transfers and accounts for kernel execution interference via an adaptive prediction model. Using these predictions, Strait performs priority-aware scheduling to deliver differentiated service between high- and low-priority requests. Evaluations under dense workloads show that Strait reduces deadline violation rates for high-priority tasks by 1.02 to 11.18 percentage points while imposing an acceptable cost on low-priority tasks. Compared with software-defined preemption approaches, Strait also demonstrates more balanced overall performance. The paper (arXiv:2604.28175) is authored by Sangjin Jung, Seonghak Hong, Insu Jang, and colleagues, and falls in the ML/systems research area.

Paper Overview

Research Area: ML/System Authors: Sangjin Jung, Seonghak Hong, Insu Jang et al. Published: 2026-04-30 arXiv: 2604.28175

Abstract

Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios.

We present Strait, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfers and accounts for kernel execution interference through an adaptive prediction model. Based on these predictions, Strait performs priority-aware scheduling to enable differentiated handling of requests.

Key Results

Under dense workloads, Strait reduces the deadline violation rate for high-priority tasks by 1.02 to 11.18 percentage points.
The improvement comes at an acceptable cost to low-priority tasks.
Compared with software-defined preemption approaches, Strait exhibits more balanced performance.

---

*Auto-collected on 2026-05-02*