Filtered by: LLM × Clear all

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Fabian Waschkowski, Prabod Rathnayaka, Lukas Wesemann 2026-07-23
Attention Inference Language Model LLM × Accelerator GPU Hardware

The paper addresses the problem of maximizing LLM inference throughput on Apple M5 hardware by leveraging its redesigned GPU with dedicated Neural Accelerators. The method, BaseRT, is a native Metal inference runtime that introduces hand-written Metal 4 tensor-core kernels for dense and mixture-of-experts GEMM and flash-attention prefill, routing compute-bound operations through the M5 Neural Accelerators while keeping memory-bound decode on existing kernels. On an M5 Pro across fifteen model configurations from Qwen3, Llama 3.2, and Gemma 4 families, BaseRT achieves up to 6.4× higher prompt-processing throughput than llama.cpp and 3.9× higher than MLX, with decode improvements up to 1.75× and 1.33× respectively. This establishes a new performance ceiling for on-device LLM inference, demonstrating that M5 tensor cores are decisive for prompt processing on Apple Silicon.

PDF

HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

Hakbeom Jang, Inho Song, Sam H. Noh, Jongryool Kim 2026-07-23
Prefetching Microarchitecture Simulation Cache LLM × Serving

HyMCache addresses the problem of high memory costs in multi-turn LLM serving by proposing a KV-cache framework that uses CXL-hybrid memory (CXL-HM), combining a small in-device DRAM with large SSD-backed capacity. The method employs request-level prefix prefetching and opportunistic write buffering to stage latency-critical reads in device DRAM, exploiting the read-dominant and predictable access patterns of multi-turn KV-cache. Experimental evidence on a real CXL-HM prototype shows HyMCache outperforms local LMCache by 3.0x in single-node and 1.45x in PD-disaggregated serving under the same DRAM budget, and incurs about 30% lower performance than 1 TB distributed-DRAM Mooncake while using 16x less DRAM. This matters because it enables TB-scale shared KV-cache reuse at SSD-level cost, significantly reducing DRAM requirements for scalable multi-turn LLM serving.

PDF