Strait: Priority-Aware, Interference-Aware Scheduling for Large-Scale ML Inference

Forum topic · 小凯 · 2026-05-03

Summary

This zhichai.net forum post offers a Feynman-style explanation of Strait (2026.05), a system paper on machine learning inference serving. The author contrasts traditional FIFO scheduling—likened to a rigid toll booth where long-context batch jobs and latency-sensitive VIP requests compete on equal footing—with Strait's two-layer approach. First, Strait perceives request priority via business logic, dynamically promoting latency-critical requests the way an ambulance uses an emergency lane. Second, it models resource interference inside GPUs: co-batching tasks that contend for the same memory bandwidth causes what the author calls compute stomping, so Strait predicts hardware-level interference and co-locates complementary workloads (e.g., CPU-bound with memory-bound) like a seating chart. The post's core takeaway is that high concurrency is achieved not by adding capacity but by matching each task's physical resource profile within microsecond windows—infrastructure competition has shifted from buying GPUs to squeezing the last drop of silicon. Readers are advised to replace naive queues with an interference-aware matrix when optimizing high-concurrency systems.

This post is a Feynman-style walkthrough of Strait (2026.05), a systems paper on machine learning inference serving. The author argues that large model API clusters suffering under massive concurrent load need not just more capacity, but a smarter traffic cop.

The problem: a rigid, priority-blind toll booth

Modern serving clusters handle billions of requests daily. Traditional schedulers behave like FIFO queues at a highway toll station: a heavy truck (long-context, compute-heavy inference) and an ambulance (latency-sensitive real-time conversation) queue in the same lane. When trucks pile up, the ambulance is stuck — the author calls this *physical congestion lacking perception dimensions*.

Strait: a traffic cop with X-ray vision

Strait's logic: don't just add lanes — instantly see each vehicle's importance and resource impact. It rebuilds scheduling at two levels:

Perceiving priority: the system tags requests dynamically based on business logic. Short VIP requests get promoted, like an ambulance taking the emergency lane past the trucks.
Perceiving interference: inside a GPU, co-batching two tasks that both hammer memory bandwidth causes them to fight over physical resources ("compute stomping"). Strait predicts this physical repulsion in advance and seats workloads like a dinner chart — pairing CPU-hungry tasks with memory-hungry ones so two big eaters never share a table. The author calls this *micro-scheduling based on hardware affinity*.

The Feynman-style judgment

High concurrency is not about flattening tasks into machines. It is about inspecting each computation's physical character within microsecond windows and meshing them, like gears, onto the GPU's finite silicon resources.

The takeaway: the AI infrastructure race has moved from *buying GPUs* to *squeezing the last drop of silicon*. When a scheduler can navigate between macro-level business priorities and micro-level memory contention, we truly tame the compute monster.

Takeaways

Don't optimize high-concurrency systems with naive first-come-first-served queues.
Build a resource interference matrix for your workloads.
If you can't tell whether incoming tasks are lions or sheep, putting them in the same cage (batch) will cause a system-wide compute collapse.