Filtered by: Microarchitecture × Prefetching × Clear all

CW-Ghost: Search-Free Granularity Selection for Helper-Thread Prefetching via Capacity Windows

Ya Zhang, Tong Lei, Yao Chen, Yonggang Che 2026-07-30
Prefetching × Microarchitecture × Simulation Search

CW-Ghost addresses the problem of costly granularity selection in helper-thread prefetching by proposing a search-free method. The method uses a single offline profiling run to estimate demand cache line fill volume and combines it with a cache capacity budget to derive a Capacity Window for iteration granularity. Experimental evidence on Intel and AMD platforms shows geometric mean speedups of 1.54x and 1.33x over original programs, and 15.8% and 10.8% improvements over Ghost Threading, achieving over 99% of optimal performance. This matters because it demonstrates that cache capacity constraints can effectively and efficiently guide helper-thread prefetching granularity without exhaustive configuration search.

PDF

DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu 2026-07-30
Prefetching × Microarchitecture × Simulation LLM Inference

DualDecoder addresses the memory bottleneck in long-context LLM inference caused by the growing KV cache. It predicts critical KV entries for the next token from the preceding speculated token, enabling proactive prefetching from host memory. Experiments show up to 2.62× throughput improvement over state-of-the-art systems without degrading latency or model quality. This matters because it eliminates GPU memory overhead from auxiliary states, making high-concurrency long-context inference more efficient for agentic applications.

PDF