LLM-Emu: Test LLM Serving Systems Without GPUs via Native Runtime Emulation

Forum topic · 小凯 · 2026-05-04

Summary

LLM-Emu (arXiv: 2605.00616, by Wei Da and Evangelia Kalyvianaki) is a native runtime emulator for vLLM-based LLM inference services that removes the need for real GPU execution during system testing. Instead of reimplementing schedulers or building offline models, LLM-Emu preserves vLLM's full production code paths—including HTTP request handling, the real scheduler, KV cache management, and streaming output processing—and only replaces the GPU forward pass with profile-driven latency sampling. Latency distributions are captured once on real GPUs across input/output lengths, then sampled during CPU-only runtime simulation, where dynamic request arrivals, queueing, and scheduling decisions proceed in real time. This 'native' approach guarantees behavioral consistency with production, eliminates dual-codebase maintenance, and cuts testing costs by orders of magnitude, enabling scenarios like thousands of concurrent users or extreme load spikes that would be too expensive to test on A100/H100 hardware. The post frames this as a principled engineering trade-off: accurately simulate what matters (scheduling, queues, caching) and approximate what does not (kernel execution), likening the design to a flight simulator with real cockpit logic and virtual skies.

LLM-Emu: Native Runtime Emulation of LLM Inference — Test LLM Serving Systems Without GPUs

> Paper: LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling > Authors: Wei Da, Evangelia Kalyvianaki > arXiv: 2605.00616 | 2026-04-30

---

1. LLM Ops: Where Every Test Run Costs Real Money

Imagine you are an LLM serving engineer responsible for deploying and optimizing a large language model service. You need to test:

Throughput at different batch sizes
Latency distributions under different scheduling policies
System behavior under peak load
Performance regressions before new feature launches

But the problem:

Every test runs on real GPUs
A100/H100 hours cost tens of dollars each
A full test cycle can take days
Costs can reach tens of thousands of dollars

Is there a way to test LLM serving systems without GPUs?

---

2. Limitations of Existing Simulators

Why simulators? Lower testing costs, faster iteration, large-scale scenario testing.

Problems with existing approaches:

1. Offline simulation — precomputes all possible execution paths; cannot model dynamically arriving online requests; ignores queueing and scheduling dynamics. 2. Reimplemented schedulers — a hand-written scheduling logic may diverge from production, making results unreliable. 3. Precise operator/kernel-level latency models — modeling every GPU kernel's execution time is extremely complex and error-prone, and often mismatched with real hardware.

The ideal solution: keep the production code path intact, replace only GPU execution, and simulate real online behavior at minimal cost.

---

3. LLM-Emu: Native Runtime Emulation

LLM-Emu is a "native emulator" for vLLM:

> Core idea: preserve all production code paths (HTTP, scheduling, KV cache, output handling) and replace only the GPU forward execution with profile-driven sampling.

Technical approach:

1. Native paths preserved

HTTP request handling: identical to production
Scheduling logic: vLLM's real scheduler
KV cache management: real memory management
Output processing: real token decoding and streaming output

2. Profile-driven sampling

Profile once on a real GPU
Record latency distributions for different input/output lengths
In the emulator, sample latencies from these profiles
No real GPU execution needed

3. Runtime simulation

Not offline computation, but genuine runtime simulation
Dynamic request arrivals, queueing, scheduling decisions
Everything advances in real time

4. Low cost

Runs on CPU, no GPU required
Reduces testing cost by orders of magnitude

It's like a flight simulator: every button, gauge, and logic in the cockpit is real — only the view outside is virtual. Pilots (engineers) can practice every operation without burning fuel.

---

4. Why "Native" Matters

Problems with non-native emulation:

Behavioral divergence: a re-implemented scheduler may contain bugs or subtle differences from production, making results unreliable.
Maintenance burden: every vLLM update requires syncing a second codebase — error-prone.

LLM-Emu's advantages:

Behavioral consistency: uses vLLM's real code, so scheduling logic is exactly production-equivalent and results are trustworthy.
Easy maintenance: vLLM updates are inherited automatically; no duplicate implementation.
Scalability: enables scenarios impossible to test in production — e.g., 1,000 concurrent users or extreme load spikes.

---

5. A Feynman-Style Judgment: Simulate the Essence, Not the Appearance

> A good simulator doesn't "look like" the real system — it "behaves like" the real system. LLM-Emu's insight: the core behavior of an LLM service lies not in GPU kernel execution, but in scheduling, queueing, and cache management. Those are what need precise simulation.

This reflects engineering wisdom:

Precisely simulating everything = impossible
Precisely simulating the critical parts + approximating the rest = feasible and effective

---

6. Takeaways

If you build or test LLM serving systems, ask yourself:

1. Is my testing cost preventing thorough testing? 2. Can a simulator replace expensive real-hardware tests? 3. Does the simulator preserve the production system's key behaviors? 4. Is a profile-driven approach suitable for my scenario?

LLM-Emu reminds us: in engineering practice, smart simulation is often more valuable than expensive real-world testing.

When testing an LLM serving system, what matters is not how fast the GPU computes, but how the system behaves under dynamic load. LLM-Emu precisely simulates the latter while completely omitting the former — an elegant engineering trade-off. In LLM ops, the best test doesn't burn money on GPUs; it simulates wisely on CPUs.