Filtered by: Language × Clear all

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

Minyu Cui, Anna Wingkvist, Morgan Ericsson 2026-07-23
Language × Model GPU Communication Scheduling

The problem is that conventional Mixture-of-Experts (MoE) implementations launch the return all-to-all communication after expert compute completes, exposing communication latency on the critical path and reducing GPU utilization. The method is a fine-grained producer-consumer co-design using tile-level signaling and scheduling, combining a persistent per-rank computation kernel that prioritizes remote-critical tiles and a persistent communication kernel on a small dedicated SM partition for segment-granular transfers. Experimental evidence on a 4-A100 GPU platform across three MoE models shows up to 2.64x end-to-end speedup and 2.74x MoE-layer speedup over four state-of-the-art systems, with consistent performance improvements across varying GEMM shapes and router modes. This matters because it provides a practical, non-intrusive approach to overlapping expert compute with the second all-to-all, significantly improving distributed MoE execution efficiency for scaling LLMs to trillion-parameter regimes.

PDF

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Fabian Waschkowski, Prabod Rathnayaka, Lukas Wesemann 2026-07-23
Attention Inference Language × Model LLM Accelerator GPU Hardware

The paper addresses the problem of maximizing LLM inference throughput on Apple M5 hardware by leveraging its redesigned GPU with dedicated Neural Accelerators. The method, BaseRT, is a native Metal inference runtime that introduces hand-written Metal 4 tensor-core kernels for dense and mixture-of-experts GEMM and flash-attention prefill, routing compute-bound operations through the M5 Neural Accelerators while keeping memory-bound decode on existing kernels. On an M5 Pro across fifteen model configurations from Qwen3, Llama 3.2, and Gemma 4 families, BaseRT achieves up to 6.4× higher prompt-processing throughput than llama.cpp and 3.9× higher than MLX, with decode improvements up to 1.75× and 1.33× respectively. This establishes a new performance ceiling for on-device LLM inference, demonstrating that M5 tensor cores are decisive for prompt processing on Apple Silicon.

PDF