The conventional wisdom in LLM inference is that latency grows smoothly as generation length increases. On the Apple MPS backend, that assumption breaks down.
Hendria (arXiv:2605.08913) discovered non-monotonic latency behavior on Apple MPS — within certain decoding budget intervals, latency suddenly spikes by up to 21x, then recovers at nearby configurations. This phenomenon is completely absent on CPU and NVIDIA CUDA backends.
The study finds a strong interaction between the KV cache and these anomalous execution regimes: inside the anomalous intervals, the KV cache speedup almost disappears. Even with the KV cache disabled, the non-monotonic behavior persists. This indicates that autoregressive decoding on the MPS backend enters discrete execution states — states that coarse-grained benchmarks cannot capture.
What remains unclear: the author speculates this may stem from the MPS backend's GPU kernel scheduling strategy, but has not pinpointed a driver-level cause. Another open question is whether new degradation patterns emerge in non-monotonic behavior over longer context windows.
---
References
1. Hendria, W. F. (2026). *Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes*. arXiv:2605.08913 [cs.LG]. 2. Apple. (2024). *Metal Performance Shaders: GPU-Accelerated ML Frameworks*. 3. Jouppi, N., et al. (2023). *TPU v4: An Optically Reconfigurable Supercomputer*. ISCA 2023.