HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

HyMCache addresses the problem of high memory costs in multi-turn LLM serving by proposing a KV-cache framework that uses CXL-hybrid memory (CXL-HM), combining a small in-device DRAM with large SSD-backed capacity. The method exploits the read-dominant, predictable, and append-only nature of multi-turn KV-cache access, using request-level prefix prefetching and opportunistic write buffering to stage latency-critical reads in device DRAM. Experimental evidence on a real CXL-HM prototype shows that under the same DRAM budget, HyMCache outperforms local LMCache by 3.0x in single-node serving and 1.45x in PD-disaggregated serving, and compared to 1 TB distributed-DRAM Mooncake, it incurs about 30% lower performance but uses 16x less DRAM. This matters because it enables TB-scale SSD-backed KV reuse at DRAM-scale efficiency and SSD-level cost, significantly reducing memory expenses for long-context and agentic LLM workloads.