GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

Problem: Low-bit quantization for LLM inference struggles when combining rotation and fine-grained group quantization due to a mismatch between global rotation and localized group scaling, causing accuracy loss or hardware overhead. Method: GyRot proposes an algorithm-hardware co-design framework with Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) to integrate rotation and group quantization, plus a zero-point rounding strategy for fully integer dequantization. Finding: On an INT4-based tensor PE architecture, GyRot achieves state-of-the-art 4-bit accuracy across LLaMA-family models, delivering up to 3.4x speedup and 3.6x energy efficiency over baseline LLM accelerators. Why it matters: This validates GyRot's practical effectiveness for scalable and energy-efficient LLM deployment, addressing a critical bottleneck in low-bit inference hardware.