Filtered by: Inference × Clear all

WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

Haozhe Hu, Hao Wu, Peiran Yin, Chao Han 2026-07-31
Attention Inference × LLM Training Hardware

WIDE introduces the first end-to-end differentiable token-level dynamic width pruning framework for LLMs, enabling fine-grained computation allocation by letting each token select attention-head and FFN-channel groups. The method uses a two-stage training pipeline and a pruning–kernel co-design that decomposes acceleration into mask reordering and block-level skipping for practical execution. At 50% sparsity, WIDE achieves a 55.1% performance boost over state-of-the-art dynamic depth pruning in calibration-only settings, with kernel-level speedups up to 1.98x for prefill and 4.95x for decoding, and end-to-end accelerations of 1.68x and 1.55x. This matters because it makes fine-grained dynamic pruning hardware-efficient, closing the gap between accuracy retention and real-world inference speedups for LLMs.

PDF

GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

Sangjin Kim, Yuseon Choi, Byeongcheol Kim, Jungjun Oh 2026-07-31
Inference × LLM Quantization Accelerator Hardware Low Bit Tensor

Problem: Low-bit quantization for LLM inference struggles when combining rotation and fine-grained group quantization due to a mismatch between global rotation and localized group scaling, causing accuracy loss or hardware overhead. Method: GyRot proposes an algorithm-hardware co-design framework with Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) to integrate rotation and group quantization, plus a zero-point rounding strategy for fully integer dequantization. Finding: On an INT4-based tensor PE architecture, GyRot achieves state-of-the-art 4-bit accuracy across LLaMA-family models, delivering up to 3.4x speedup and 3.6x energy efficiency over baseline LLM accelerators. Why it matters: This validates GyRot's practical effectiveness for scalable and energy-efficient LLM deployment, addressing a critical bottleneck in low-bit inference hardware.

PDF

A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference

Jing Ding, Yash Nishant, Chandrish Ambati, Jyothsna Kamati 2026-07-31
Cache LLM Inference × GPU Architecture Simulation Workload

The paper addresses the memory wall in LLM inference, where KV cache demands tens of terabytes at hundreds of GB/s exceed current memory tier capabilities. The proposed Marvell Photonic Fabric Memory Appliance replaces electrical switches with a passive fiber shuffle in a switch-free full-crossbar topology, delivering 32 TB shared memory across 16 hosts via photonic-CXL hybrid architecture. Emulation results show over 50% latency reduction versus electrical CXL pools, while simulation demonstrates a 6.6x improvement in time-to-first-token by eliminating cache eviction cliffs for multi-turn workloads. This work matters because it enables practical TB-scale shared memory for concurrent long-context users, overcoming the scalability limits of electrical CXL pooling in real deployments.

PDF