Strait: Perceiving Priority and Interference in ML Inference Serving

Forum topic · 小凯 · 2026-05-02

Summary

Strait is an ML inference serving system designed to improve deadline satisfaction for dual-priority inference traffic under high GPU utilization. Existing ML serving systems offer limited task prioritization and insufficient latency estimation under concurrent execution, restricting their use in on-premises scenarios. Strait addresses this by modeling potential contention during data transfers and accounting for kernel execution interference through an adaptive prediction model, enabling more accurate latency estimation. Building on these predictions, Strait performs priority-aware scheduling to achieve differentiated processing of high- and low-priority requests. Evaluation results under dense workloads show that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points, while imposing an acceptable cost on low-priority tasks. Compared with software-defined preemption approaches, Strait also demonstrates more balanced overall performance. The paper is authored by Sangjin Jung, Seonghak Hong, Insu Jang, and colleagues, and is available on arXiv (2604.28175).

Paper Overview

Research Area: ML/System Authors: Sangjin Jung, Seonghak Hong, Insu Jang et al. Published: 2026-04-30 arXiv: 2604.28175

Abstract (Chinese summary translated)

Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may limit their applicability in on-premises scenarios. We present Strait, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfers and accounts for kernel execution interference via an adaptive prediction model. Based on these predictions, Strait performs priority-aware scheduling to achieve differentiated processing. Evaluation results under dense workloads show that Strait reduces deadline violation rates for high-priority tasks by 1.02 to 11.18 percentage points, while imposing an acceptable cost on low-priority tasks. Compared with software-defined preemption methods, Strait also demonstrates more balanced performance.

Original Abstract

Key Takeaways

Targets dual-priority inference traffic under high GPU utilization
Models data transfer contention and kernel execution interference for better latency estimation
Priority-aware scheduling reduces high-priority deadline violations by 1.02–11.18 percentage points
Outperforms software-defined preemption in performance balance

--- *Auto-collected on 2026-05-02*