KVMem (kvmem/kvmem-llama.cpp) is a new approach built on llama.cpp: run Qwen3.8-27B on a 16GB VRAM card with the full 256K context, decoding at a stable 30-40 tok/s (benchmarked on an RTX 5060 Ti). The idea in one sentence: completed historical KV cache is moved into system memory as a pool, the GPU keeps only a limited activation window, and at each step only the parts most relevant to the current query are fetched back into VRAM — a 256K workspace with only 32K resident in GPU memory. The project was open-sourced on September 14, and is accompanied by an arXiv paper (2609.04852, *Virtualizing Million-Token Agent Workspaces on a Consumer GPU*). On a 24GB RTX 5090 Laptop, the workspace can extend to 1M tokens. The accuracy trade-off: on LongMemEval-S / AgentLongBench, a 32K GPU window against the full 256K history scores 85.6% vs 86.6% accuracy and 60.9% vs 59.5% task success — nearly lossless. It exposes an OpenAI-compatible interface with chat/tools/vision support, so existing local clients can point directly at it.
Below are the release notes:
---
Repository: https://github.com/kvmem/kvmem-llama.cpp
Recommended review video by creator melis, with real end-to-end task examples: https://www.bilibili.com/video/BV11deu6xEdD/
Windows CUDA 13/12 prebuilt binaries: https://github.com/kvmem/kvmem-llama.cpp/releases/tag/v0.16.0-rc3 Windows ROCm prebuilt binaries: https://github.com/kvmem/kvmem-llama.cpp/releases/tag/rc3-rocm-beta
The Windows CUDA 13 build theoretically supports NVIDIA RTX 20/30/40/50 series consumer cards, Quadro RTX, RTX A, RTX Ada, and RTX PRO Blackwell professional cards, plus server GPUs such as T10, T4, A10, A30, A40, A100, L4, L40/L40S, and H100/H200. The CUDA 12 build additionally supports V100.
ROCm HIP includes gfx1100/gfx1200/gfx1201, theoretically supporting RX 7900 series / 9060 XT / 9070 series.
Please report any installation or usage issues. Thank you.
Note: a Mac port is in development (Mac will offload KV to disk to free unified memory, e.g., letting a 48GB MacBook Pro run large contexts more memory-efficiently).
Feedback via issues and trial experiences are welcome!
---
Editor's verification notes (checked item by item before publication):
The repository and releases have both been verified: v0.16.0-rc3 (09-20, including Windows CUDA 13/12 packages) and rc3-rocm-beta (09-23, Windows/Linux packages) exist, followed by rc3-rocm-beta2 (09-24) and v0.16.0-rc3-prism.3 (09-24, with CODING-MTP-RESULTS.md) — iteration is fast; check the latest tag before installing. The repository was created 09-14 and reached 700+ stars within a week.
Two things the original post does not expand on, but which matter for deciding whether to install:
First, the lossy nature is openly acknowledged. Community members have raised objections (citing the CacheBlend paper, arguing that standard-attention KV cannot be chunked this way). The author's verbatim response: "I know we are not computationally equivalent to the original kv, and we did not try to prove that. But this lossy retrieval works very well in practice — actually better than we expected." The 85.6% vs 86.6% figures above are measured boundaries, not a losslessness guarantee. The video creator added a rule of thumb: the activation window can support roughly 6x its length in logical context; with the default 32K activation, pushing logical length beyond ~200K tends to cause hallucinations — if you want stability, don't max it out.
Second, GPU compatibility is stated as "theoretical". The notes say "architecture code is included; specific models require verification against drivers and real hardware." The community's实测 benchmark table compiled over three days (5060 Ti 16G at 30-40 t/s, 5070 Ti 16G at 50-65, 4070 TiS at 40-70, 5050 laptop prefill 100+/decode 10, an 8GB card running ternary models at 22 t/s under 128K) mostly covers 40/50-series consumer cards; feedback on 20-series and server cards is still thin — check the issue tracker before buying hardware.
If Windows deployment shows significant slowdown, check whether VRAM usage is at 100%: (1) offload the multimodal head to CPU (default in the latest rc3); (2) if that fails, reduce budget to 32K and gen reserve to 10K.