[论文] TideCache: Efficient Hybrid LLM Serving with Transparent Cache Managem...
研究领域: ML 作者: Jinbin Luo, Xiaozhe Yao, Shang Yang, Zhijian Liu, Song Han, Qinghao Hu 发布时间: 2026-11-19 arXiv: 2611.26619
论文概要
研究领域: ML 作者: Jinbin Luo, Xiaozhe Yao, Shang Yang, Zhijian Liu, Song Han, Qinghao Hu 发布时间: 2026-11-19 arXiv: 2611.26619
中文摘要
混合架构已成为开源权重语言模型的主流。其全局注意力层和有界状态层维护异构缓存,需求随上下文长度和并发度变化。我们提出TideCache,一个部署在CPU-GPU超级芯片上的 serving 系统,包含三个组件用于在缓存间分配GPU内存并管理其跨GPU与主机内存的驻留。透明缓存池在固定虚拟地址后重新分配物理页,保持缓存布局和后端接口不变。缓存流式器利用有界状态层提供的计算间隔,将下一全局注意力层的前缀预取到跨全局注意力层共享的GPU槽位中,降低GPU驻留需求并将逐层传输与计算重叠。生成KV保持驻留。请求调度器根据请求大小和队列状态选择驻留或流式执行,并触发容量调整。三者协同使流式释放的HBM可容纳其他缓存组件和更多请求。在SGLang中实现并在GB300和GH200上评估,跨七种模型/平台配置,TideCache完成多达2.04倍的智能体回合数,将整体P50每token延迟降低20-59%,并将智能体P50 TTFT提升1.40-3.05倍。
原文摘要
Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU-GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers ...
*自动采集于 2026-11-21*
#论文 #arXiv #ML #小凯