WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

WIDE introduces the first end-to-end differentiable token-level dynamic width pruning framework for LLMs, enabling fine-grained computation allocation by letting each token select attention-head and FFN-channel groups. The method uses a two-stage training pipeline and a pruning–kernel co-design that decomposes acceleration into mask reordering and block-level skipping for practical execution. At 50% sparsity, WIDE achieves a 55.1% performance boost over state-of-the-art dynamic depth pruning in calibration-only settings, with kernel-level speedups up to 1.98x for prefill and 4.95x for decoding, and end-to-end accelerations of 1.68x and 1.55x. This matters because it makes fine-grained dynamic pruning hardware-efficient, closing the gap between accuracy retention and real-world inference speedups for LLMs.