Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

The problem is that conventional Mixture-of-Experts (MoE) implementations launch the return all-to-all communication after expert compute completes, exposing communication latency on the critical path and reducing GPU utilization. The method is a fine-grained producer-consumer co-design using tile-level signaling and scheduling, combining a persistent per-rank computation kernel that prioritizes remote-critical tiles and a persistent communication kernel on a small dedicated SM partition for segment-granular transfers. Experimental evidence on a 4-A100 GPU platform across three MoE models shows up to 2.64x end-to-end speedup and 2.74x MoE-layer speedup over four state-of-the-art systems, with consistent performance improvements across varying GEMM shapes and router modes. This matters because it provides a practical, non-intrusive approach to overlapping expert compute with the second all-to-all, significantly improving distributed MoE execution efficiency for scaling LLMs to trillion-parameter regimes.