Filtered by: LLM × Clear all

Harness Engineering for LLM-Driven GPU Kernel Generation

Yue Shui, Chenyu Ma, Hangfei Xu, Shengzhao Wen 2026-07-22
Language Model LLM × GPU

LLM-driven GPU kernel generation is unreliable without a system to constrain, validate, and select candidate code. This paper introduces a harness-centered system that separates an evaluation harness from a profile-backed optimization controller to enforce compilation, correctness, and timing. Across five operator definitions, the system achieved mean-latency speedups over FlashInfer baselines ranging from 1.12x to 29.68x on NVIDIA Blackwell B200 GPUs. The findings demonstrate that expert-provided optimization directions and workload context remain critical for reliable AI-driven kernel optimization.

PDF

ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels

Bingyang Wu, Chao Jin, Zili Zhang, Xinming Wei 2026-07-22
Attention Fp8 GPU Serving LLM ×

ExpertPlex addresses the problem of inefficient resource utilization in serving large Mixture-of-Experts (MoE) LLMs, where existing prefill-decode disaggregation wastes resources due to coarse allocation and colocation suffers from phase interference and poor load tracking. The method shares massive MoE experts across phases while disaggregating attention modules, and introduces adaptive persistent kernels, attention-initiated MoE communication, and a tile-to-cluster model to optimize execution. Experiments serving MiniMax-M2.7 and GLM-5.1-FP8 show ExpertPlex improves goodput by up to 2.01× over instance-level disaggregation and 1.66× over colocation. This matters because it enables efficient, high-throughput serving of increasingly large MoE models without overprovisioning or phase interference.

PDF

HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

Hakbeom Jang, Inho Song, Sam H. Noh, Jongryool Kim 2026-07-22
Prefetching Microarchitecture Simulation Cache LLM × Serving

HyMCache addresses the problem of high memory costs in multi-turn LLM serving by proposing a KV-cache framework that uses CXL-hybrid memory (CXL-HM), combining a small in-device DRAM with large SSD-backed capacity. The method exploits the read-dominant, predictable, and append-only nature of multi-turn KV-cache access, using request-level prefix prefetching and opportunistic write buffering to stage latency-critical reads in device DRAM. Experimental evidence on a real CXL-HM prototype shows that under the same DRAM budget, HyMCache outperforms local LMCache by 3.0x in single-node serving and 1.45x in PD-disaggregated serving, and compared to 1 TB distributed-DRAM Mooncake, it incurs about 30% lower performance but uses 16x less DRAM. This matters because it enables TB-scale SSD-backed KV reuse at DRAM-scale efficiency and SSD-level cost, significantly reducing memory expenses for long-context and agentic LLM workloads.

PDF