InferScale: GPU-Native KV Injection for Personalized LLM Serving

InferScale addresses the problem of increasing time-to-first-token (TTFT) in personalized LLM serving due to repeated prefilling of persistent user memory. The method precomputes KV representations for each memory fact, stores them on the GPU with semantic embeddings, and injects them directly into vLLM's paged cache using Chunked RoPE and Context-Window Encoding. On LoCoMo with three open-weight models, InferScale reduces TTFT by 72-79% (3.6-4.8x) at k=50, achieves 60.3% accuracy versus 63.3% for Mem0, and delivers 3.7-4.5x throughput under concurrent load. This matters because reusable KV state decouples memory-conditioned serving latency from retrieved-context size, enabling scalable personalized LLM serving without engine modifications or fine-tuning.