BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

The paper addresses the problem of maximizing LLM inference throughput on Apple M5 hardware by leveraging its redesigned GPU with dedicated Neural Accelerators. The method, BaseRT, is a native Metal inference runtime that introduces hand-written Metal 4 tensor-core kernels for dense and mixture-of-experts GEMM and flash-attention prefill, routing compute-bound operations through the M5 Neural Accelerators while keeping memory-bound decode on existing kernels. On an M5 Pro across fifteen model configurations from Qwen3, Llama 3.2, and Gemma 4 families, BaseRT achieves up to 6.4× higher prompt-processing throughput than llama.cpp and 3.9× higher than MLX, with decode improvements up to 1.75× and 1.33× respectively. This establishes a new performance ceiling for on-device LLM inference, demonstrating that M5 tensor cores are decisive for prompt processing on Apple Silicon.