KVMem (kvmem/kvmem-llama.cpp) is a new approach built on llama.cpp: a 16GB GPU can run Qwen3.8-27B with a full 256K context at a stable 30-40 tok/s during decode (benchmarked on an RTX 5060 Ti). The idea in one sentence: move the KV cache of completed history into system memory as a pool, keep only a limited activation window on the GPU, and retrieve just the parts most relevant to the current query—so a 256K workspace only keeps 32K resident in VRAM. The project was open-sourced on September 14, with an arXiv paper (2609.04852, *Virtualizing Million-Token Agent Workspaces on a Consumer GPU*). On a 24GB RTX 5090 Laptop, the workspace can extend to 1M tokens. Accuracy: on LongMemEval-S / AgentLongBench, a 32K GPU window versus the full 256K history gives 85.6% vs 86.6% accuracy and 60.9% vs 59.5% task success rate—essentially lossless.
Release notes:
---
Repo: https://github.com/kvmem/kvmem-llama.cpp
Recommended review video by uploader melis, with real task examples: https://www.bilibili.com/video/BV11deu6xEdD/
Windows CUDA 13/12 prebuilt builds: https://github.com/kvmem/kvmem-llama.cpp/releases/tag/v0.16.0-rc3 Windows ROCm prebuilt builds: https://github.com/kvmem/kvmem-llama.cpp/releases/tag/rc3-rocm-beta
The Windows CUDA 13 build should support NVIDIA RTX 20/30/40/50-series consumer cards, Quadro RTX, RTX A, RTX Ada, and RTX PRO Blackwell professional cards, plus server GPUs such as T10, T4, A10, A30, A40, A100, L4, L40/L40S, and H100/H200. The CUDA 12 build also supports V100.
ROCm HIP includes gfx1100/gfx1200/gfx1201, nominally supporting RX 7900 series / 9060 XT / 9070 series.
Please report any installation or usage issues.
Note: a Mac port is in development (Mac will offload KV to disk to free unified memory, e.g. letting a 48GB MacBook Pro run larger contexts more efficiently).
Issues and trial feedback welcome!
---
Editor's verification notes (checked item by item before publishing):
The repo and releases are verified: v0.16.0-rc3 (09-20, with Windows CUDA 13/12 packages) and rc3-rocm-beta (09-23, Windows/Linux) are live, followed by rc3-rocm-beta2 (09-24) and v0.16.0-rc3-prism.3 (09-24, with CODING-MTP-RESULTS.md)—iteration is fast, so check the latest tag before installing. The repo was created 09-14 and gained 700+ stars in a week.
Two things the original post didn't expand on, but that matter for deciding whether to install:
1. The lossiness is out in the open. There is community skepticism (citing the CacheBlend paper, arguing standard-attention KV can't be chunked this way). The author's verbatim response: "I know we are not computationally equivalent to the original KV, and we did not try to prove that. But this lossy recovery works very well in practice—in fact better than we expected." The 85.6% vs 86.6% figures above are the measured boundary, not a lossless guarantee. The video uploader added a rule of thumb: the activation window supports roughly 6x its length in logical context; pushing logical length beyond ~200K with the default 32K activation window tends to cause hallucinations—don't max it out if you want stability.
2. GPU support is stated as "theoretical." The notes say "the corresponding architecture code is included; specific models need driver and real-hardware validation." Community benchmarks gathered over three days (5060 Ti 16G at 30-40 t/s, 5070 Ti 16G at 50-65, 4070 TiS at 40-70, 5050 laptop with prefill 100+/decode 10, and an 8GB card running ternary models at 22 t/s with 128K context) mostly cover 40/50-series consumer cards; there is little feedback on 20-series and server GPUs—check the issue tracker before buying.
If Windows deployment shows significant slowdown, check whether VRAM usage is at 100%: (1) offload the multimodal head to CPU (default in the latest rc3); (2) if that doesn't help, reduce budget to 32K and gen reserve to 10K.