InferScale: GPU-Native KV Injection for Personalized LLM Serving

https://arxiv.org/abs/2607.27090v1

Core Idea

InferScale is a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state, addressing the TTFT increase caused by injecting persistent personalized context into prompts.

For this daily profile, it is worth opening because it links Memory, Microarchitecture, and Simulation to a concrete method, not just a broad trend.

What Is New

The novelty signal is concentrated around Memory, Microarchitecture, Simulation, and GPU. For this profile, the important question is whether the paper changes how architecture ideas are generated, evaluated, or connected to software and hardware constraints.

Methodology

Read this as a loop: define the target system, apply the proposed mechanism, measure against a baseline, then use the measured signal to justify the next design choice. Mechanism: Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Evidence: Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without.

score(design) = quality_metric(design) - cost_to_evaluate(design) + feedback_gain(design)

Figure To Read First

Read this visual first: focus on the first architecture, workflow, or pipeline figure before the experiments. It should show what is optimized, what feedback signal is used, and where the system boundary sits.

Minimal Mental Model

research artifact
  question      -> what design, runtime, or system boundary changes?
  mechanism     -> model, agent, compiler, simulator, or hardware feedback
  evaluation    -> baseline comparison plus cost / latency / accuracy signal
  reusable idea -> what should carry into the next architecture experiment?

Why It Matters

Paper recommendations matter when they sharpen the research map: what problem is now easier to study, what methodology becomes reusable, and which architecture assumptions should be questioned next.