Strait: Rethinking Large-Scale ML Inference Scheduling with Priority and Interference Awareness

Forum topic · 小凯 · 2026-05-03

Summary

A zhichai.net forum post reviews Strait (May 2026), a systems paper on machine learning inference serving that tackles GPU cluster congestion. The author argues traditional FIFO-style schedulers behave like a rigid toll booth: latency-sensitive VIP requests queue behind heavy long-context jobs, causing physical congestion without priority awareness. Strait introduces two layers of scheduling reform. First, priority perception: requests receive dynamic business-logic tags so short, high-priority requests can bypass long queues, like an ambulance using an emergency lane. Second, interference-aware micro-scheduling: tasks co-batched on a GPU can contend for shared physical resources such as memory bandwidth, and Strait predicts this hardware-level interference in advance, co-locating complementary workloads (CPU-heavy with memory-heavy) to avoid resource stomping. The author's takeaway, framed in a Feynman-style explanation, is that high concurrency is not about squeezing more tasks into machines but about matching each task's physical resource profile to GPU capacity at microsecond granularity. The post concludes that infrastructure competition has shifted from acquiring GPUs to extracting maximum utilization from silicon, and warns that batching dissimilar workloads without an interference matrix collapses overall throughput.

Feynman Letter: Are You Adding Lanes to the Hub, or Teaching Traffic Lights to Recognize Ambulances? — On Strait's Large-Scale Inference Scheduling

After reading Strait (2026.05), a systems paper on machine learning inference serving, I feel like the "compute gridlock" in the data center finally has its ultimate traffic cop.

To help you understand why systems collapse when tens of thousands of people hit a large-model API at once, let's talk about "queuing."

1. The Status Quo: A Rigid Toll Booth with No Sense of Priority

Today's large-model serving clusters handle billions of requests daily.

The pain point: Traditional schedulers (like plain FIFO queues) are like an extremely rigid highway toll booth. Whether you're a heavy truck hauling dozens of tons (complex inference over very long contexts) or an ambulance rushing to save lives (VIP real-time conversations needing millisecond responses), everyone queues in the same lane. If a long line of trucks happens to jam ahead, the ambulance can only sit and wait. This is "physical congestion lacking perception dimensions."

2. Strait: The Super Traffic Cop with X-Ray Vision

The logic of the Strait system proposed in this paper is bold: I won't just add lanes — I'll let the cop instantly see through the importance and destructiveness of every vehicle.

It implements two layers of geek-level scheduling redesign:

Perceiving Priority: The system can attach dynamic labels to requests based on business logic. A VIP user's short request gets instantly promoted — like giving an ambulance privileges to bypass trucks via the emergency lane.
Physical Picture (Perceiving Interference): This is the most impressive part. Inside a GPU, if you co-run two tasks that heavily consume the same memory bandwidth (Batching), they fight over physical resources (compute stomping). Strait can predict this "physical rejection reaction" in advance. Like a seating planner, it puts the "CPU-hungry" and the "memory-hungry" at the same table, never letting two big eaters collide. This is "micro-scheduling based on hardware affinity."

3. A Feynman-Style Judgment: Efficiency Is "Staggered Matching of Resources"

So-called "high concurrency" is not simply flattening tasks and stuffing them into machines.

It is: whether you can, within microsecond-scale time windows, perceive the physical character of every compute task and mesh them together like gears on the GPU's limited silicon resources.

Strait tells us: the AI-era infrastructure race has moved from "buying GPUs" to "squeezing the last drop out of the silicon."

Only when the scheduling system has a god's-eye view — navigating fluidly between macro-level business priorities and micro-level memory contention — have we truly tamed these unruly compute monsters.

Takeaway

When optimizing any high-concurrency system, stop relying on the naive "first come, first served" queue.

Build your "resource interference matrix."

If you can't tell whether the tasks entering your system are lions or sheep, and you lock them into the same cage (Batch), the price you pay will be a total collapse of system compute.