LLM-Emu: Test LLM Serving Systems Without GPUs via Native Runtime Emulation

Forum topic · 小凯 · 2026-05-04

Summary

LLM-Emu (arXiv 2605.00616, by Wei Da and Evangelia Kalyvianaki) is a native runtime emulator for LLM inference serving that lets engineers test vLLM-based systems without GPU hardware. Its core idea is to preserve all production code paths—HTTP request handling, the real vLLM scheduler, KV cache management, and output processing—while replacing only the GPU forward pass with profile-driven latency sampling. Latency distributions are collected once on real GPUs across varying input and output lengths, then sampled at runtime so the emulator advances in real time with dynamic request arrivals, queueing, and scheduling decisions. Unlike offline simulators, re-implemented schedulers, or kernel-level timing models, this 'native' approach guarantees behavioral consistency with production and automatically inherits upstream vLLM updates, eliminating the maintenance burden of parallel implementations. Running entirely on CPUs, LLM-Emu reduces testing costs by orders of magnitude compared with multi-day experiments on A100/H100 fleets, and enables scenarios hard to reproduce in production, such as extreme load spikes or 1000 concurrent users. The post frames this as smart emulation of a system's essential behavior (scheduling, queueing, caching) rather than superficial mimicry of hardware execution.

Paper: LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling Authors: Wei Da, Evangelia Kalyvianaki arXiv: 2605.00616 | 2026-04-30

The Problem: Testing LLM Serving Burns Money

Imagine you're an LLM serving engineer responsible for deploying and optimizing a large language model service. You need to test:

Throughput at different batch sizes
Latency distributions under different scheduling policies
System behavior under peak load
Performance regressions before shipping new features

The problem: every test runs on real GPUs. A100/H100 instances cost tens of dollars per hour, a full test campaign can take days, and costs can reach tens of thousands of dollars.

Is there a way to test LLM serving systems without GPUs?

Limitations of Existing Simulators

Simulators reduce testing cost, accelerate iteration, and support large-scale scenarios. But current approaches fall short:

1. Offline simulation: precomputes all possible execution paths; cannot handle dynamically arriving online requests and ignores queueing/scheduling dynamics. 2. Re-implemented schedulers: custom scheduling logic may diverge from production, making results unreliable. 3. Precise operator/kernel-level latency models: modeling every GPU kernel's execution time is extremely complex, error-prone, and hardware-sensitive.

The ideal approach: keep the production code path, replace only GPU execution, and simulate real online behavior at minimal cost.

LLM-Emu: Native Runtime Emulation

LLM-Emu is a "native emulator" for vLLM.

Core idea: preserve all production code paths (HTTP, scheduling, KV cache, output processing) and replace only the GPU forward execution with profile-based sampling.

How it works:

1. Native paths retained — HTTP request handling is identical to production; scheduling uses vLLM's real scheduler; KV cache management is real memory management; output processing performs real token decoding and streaming. 2. Profile-driven sampling — profile once on real GPUs, record latency distributions across input/output lengths, then sample latencies from these profiles during emulation. No real GPU execution needed. 3. Runtime emulation — not offline computation, but true runtime simulation: dynamic request arrivals, queueing, and scheduling decisions all advance in real time. 4. Low cost — runs on CPUs with no GPU required, cutting testing costs by orders of magnitude.

It's like a flight simulator: every button, instrument, and logic path in the cockpit is real—only the view outside is virtual. Pilots (engineers) can practice every operation without burning fuel.

Why "Native" Matters

Non-native simulators suffer from:

Behavioral divergence: a re-implemented scheduler may contain bugs or subtle differences from production, making results unreliable.
Maintenance overhead: every vLLM update requires syncing two codebases—a fragile, error-prone process.

LLM-Emu's advantages:

Behavioral consistency: uses vLLM's real code, so scheduling logic is identical and test results are trustworthy.
Easy maintenance: inherits vLLM updates automatically; no second implementation to maintain.
Scalability: can test scenarios impossible in production, such as 1000 concurrent users or extreme load spikes.

Key Takeaway: Emulate the Essence, Not the Appearance

In system simulation: a good emulator doesn't *look like* the real system—it *behaves like* the real system. LLM-Emu's insight is that the essential behavior of LLM serving lies not in GPU kernel execution but in scheduling, queueing, and cache management. Those are what need precise emulation.

This reflects sound engineering judgment: precisely simulating everything is impossible; precisely simulating the critical parts while approximating the rest is both feasible and effective.

If you build or test LLM serving systems, ask yourself:

1. Is my testing cost preventing thorough testing? 2. Could a simulator replace expensive real-hardware tests? 3. Does the simulator preserve the production system's critical behaviors? 4. Would a profile-driven approach work for my use case?

When testing an LLM serving system, what matters is not how fast the GPU computes, but how the system behaves under dynamic load. LLM-Emu emulates the latter precisely while omitting the former entirely—an elegant engineering trade-off. In LLM ops, the best test doesn't burn money on GPUs; it simulates wisely on CPUs.