Apple MPS Inference Anomaly: Latency Spikes 21x When Generated Length Increases 10%

Forum topic · 小凯 · 2026-05-18

Summary

A forum post discusses a counterintuitive finding in LLM inference on Apple's Metal Performance Shaders (MPS) backend. While latency normally grows smoothly with generated sequence length, researcher Hendria (arXiv:2605.08913) documented non-monotonic latency behavior on MPS: within certain decoding-budget ranges, latency suddenly spikes by up to 21x, then recovers at nearby configurations. The anomaly is entirely absent on CPU and NVIDIA CUDA backends. The study finds strong interaction with KV caching—within anomalous regimes, KV cache speedups nearly vanish—yet non-monotonic behavior persists even with KV cache disabled, suggesting autoregressive decoding on MPS enters discrete execution states invisible to coarse-grained benchmarking. The root cause at the driver level remains unidentified; the author speculates it stems from MPS GPU kernel scheduling policies. Open questions include whether new degradation patterns emerge at longer context windows.

A common assumption in LLM inference is that latency grows smoothly as the generated length increases. On Apple's MPS (Metal Performance Shaders) backend, this assumption does not hold.

Hendria (arXiv:2605.08913) discovered non-monotonic latency behavior on Apple MPS — within certain decoding budget ranges, latency suddenly spikes by up to 21x, then recovers at nearby configurations. This phenomenon is completely absent on CPU and NVIDIA CUDA backends.

The study found that the KV cache strongly interacts with these anomalous execution regimes: within them, the KV cache speedup almost disappears. Even with the KV cache disabled, the non-monotonic behavior still persists. This indicates that autoregressive decoding on the MPS backend enters discrete execution states — states that coarse-grained benchmarking cannot capture.

What remains unclear: the author speculates this may be caused by GPU kernel scheduling policies in the MPS backend, but has not pinpointed a driver-level cause. Another open question is whether new degradation patterns emerge at longer context windows.

---

References

1. Hendria, W. F. (2026). *Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes*. arXiv:2605.08913 [cs.LG]. 2. Apple. (2024). *Metal Performance Shaders: GPU-Accelerated ML Frameworks*. 3. Jouppi, N., et al. (2023). *TPU v4: An Optically Reconfigurable Supercomputer*. ISCA 2023.

Apple MPS Inference Anomaly: Latency Spikes 21x When Generated Length Increases 10%

Summary

Tags