Filtered by: Hardware × Clear all

WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

Haozhe Hu, Hao Wu, Peiran Yin, Chao Han 2026-07-31
Attention Inference LLM Training Hardware ×

WIDE introduces the first end-to-end differentiable token-level dynamic width pruning framework for LLMs, enabling fine-grained computation allocation by letting each token select attention-head and FFN-channel groups. The method uses a two-stage training pipeline and a pruning–kernel co-design that decomposes acceleration into mask reordering and block-level skipping for practical execution. At 50% sparsity, WIDE achieves a 55.1% performance boost over state-of-the-art dynamic depth pruning in calibration-only settings, with kernel-level speedups up to 1.98x for prefill and 4.95x for decoding, and end-to-end accelerations of 1.68x and 1.55x. This matters because it makes fine-grained dynamic pruning hardware-efficient, closing the gap between accuracy retention and real-world inference speedups for LLMs.

PDF

GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

Sangjin Kim, Yuseon Choi, Byeongcheol Kim, Jungjun Oh 2026-07-31
Inference LLM Quantization Accelerator Hardware × Low Bit Tensor

Problem: Low-bit quantization for LLM inference struggles when combining rotation and fine-grained group quantization due to a mismatch between global rotation and localized group scaling, causing accuracy loss or hardware overhead. Method: GyRot proposes an algorithm-hardware co-design framework with Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) to integrate rotation and group quantization, plus a zero-point rounding strategy for fully integer dequantization. Finding: On an INT4-based tensor PE architecture, GyRot achieves state-of-the-art 4-bit accuracy across LLaMA-family models, delivering up to 3.4x speedup and 3.6x energy efficiency over baseline LLM accelerators. Why it matters: This validates GyRot's practical effectiveness for scalable and energy-efficient LLM deployment, addressing a critical bottleneck in low-bit inference hardware.

PDF