Filtered by: LLM × Clear all

NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

Sookyung Choi, Seungyong Lee, Kangkyu Park, Yunseo Chun 2026-07-30
Attention LLM × Accelerator GPU Cxl Rdma Runtime Serving

NELSSA addresses the problem of GPU-centric LLM serving systems suffering from throughput and latency inefficiencies when handling highly heterogeneous mixed-length workloads. The method introduces a GPU-PNM heterogeneous system that uses length-based request placement, routing short-context requests to GPUs and long-context requests to PNM, with runtime migration for dynamic context growth. Experimental evidence shows that NELSSA improves decode throughput by up to 5.5x in tokens/sec and reduces P99 latency by up to 15x compared to GPU-only baselines across mixed-length LLM workloads. This matters because it demonstrates that integrated GPU-PNM serving, enabled by CXL-based disaggregation, is a promising paradigm for building scalable and flexible LLM infrastructures that can efficiently support evolving workloads.

PDF

The Fabric Is the Cluster Driver: Cross-Layer eBPF Policies for GPU-CXL Fabrics

Yiwei Yang, Andi Quinn 2026-07-30
Attention LLM × Compiler GPU Hardware Cxl Runtime Tensor

fabric_ext introduces a cross-layer eBPF middleware compiler and runtime for enforcing extensible OS policies across GPU-CXL fabrics. The method uses a semantic movement graph abstraction that describes data movement characteristics and is compiled into per-device eBPF programs, BPF maps, and runtime artifacts for bpftime and dputime. Experimental evidence from the LLM prefill stress case shows that fabric_ext can manage attention and FFN dataflows crossing GPU tensor execution, DPU/NIC event execution, and CXL switch-local islands. This matters because it enables unified, data-driven policy enforcement across heterogeneous fabric components, improving observability and control in disaggregated memory systems.

PDF

AgenticCANN: Automated Ascend C Operator Generation via Knowledge-Augmented Agentic Evolution

Junhao Qiu, Zidong Wang, Yansong Sun, Zhitong Ma 2026-07-30
Inference Language Model LLM × Hardware Agentic

AgenticCANN addresses the problem of automated Ascend C operator generation for NPUs, which requires deep hardware expertise and faces a severe platform knowledge deficit. The method introduces a knowledge-augmented agentic evolution framework with a knowledge-orchestrated generation system and a stage-adaptive agentic evolution strategy to dynamically align LLM interaction modes. Experiments on Huawei Ascend 910B across six operators show 90-100% feasibility on elementwise and normalization operators, 56% on fusion operators, and up to 6.65× speedup on 1B Pangu model inference kernels, with knowledge injection monotonically improving feasibility from 57% to 86%. This matters because it enables automated, high-performance operator synthesis in low-corpus NPU environments, overcoming the unique challenges of the Ascend C programming model.

PDF

DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu 2026-07-30
Prefetching Microarchitecture Simulation LLM × Inference

DualDecoder addresses the memory bottleneck in long-context LLM inference caused by the growing KV cache. It predicts critical KV entries for the next token from the preceding speculated token, enabling proactive prefetching from host memory. Experiments show up to 2.62× throughput improvement over state-of-the-art systems without degrading latency or model quality. This matters because it eliminates GPU memory overhead from auxiliary states, making high-concurrency long-context inference more efficient for agentic applications.

PDF

InferScale: GPU-Native KV Injection for Personalized LLM Serving

Peter Li, Prashant Pandey 2026-07-30
Memory Microarchitecture Simulation GPU Cache LLM × Serving

InferScale addresses the problem of increasing time-to-first-token (TTFT) in personalized LLM serving due to repeated prefilling of persistent user memory. The method precomputes KV representations for each memory fact, stores them on the GPU with semantic embeddings, and injects them directly into vLLM's paged cache using Chunked RoPE and Context-Window Encoding. On LoCoMo with three open-weight models, InferScale reduces TTFT by 72-79% (3.6-4.8x) at k=50, achieves 60.3% accuracy versus 63.3% for Mem0, and delivers 3.7-4.5x throughput under concurrent load. This matters because reusable KV state decouples memory-conditioned serving latency from retrieved-context size, enabling scalable personalized LLM serving without engine modifications or fine-tuning.

PDF

Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling

Jiaqi Yang, Jiayi Li, Yihan Fu, Hongxiao Zhao 2026-07-30
Hardware Roofline Codesign Compiler Runtime HPC LLM × Inference

The problem is that prefill-decode disaggregation and roofline-based operator placement are insufficient for partitioning LLM inference across heterogeneous systems due to workload shape, device contention, and weight layout constraints. DOPS is a hardware-aware, closed-loop framework that uses a stage-aware DAG, a Bifocal scheduler for dynamic operator placement, and a Weight Layout Arbiter (WLA) for selecting efficient weight layouts. On heterogeneous platforms combining NPUs and PIM devices, Bifocal achieves geometric-mean speedups of 1.20× to 2.23× over the PD baseline, with WLA adding 1.28× to 1.33× further speedup. This matters because DOPS enables systematic analysis of workload sensitivity and hardware scalability, improving LLM serving efficiency on diverse heterogeneous hardware.

PDF