Filtered by: Inference × Quantization × Clear all

kvcache-ai/ktransformers

kvcache-ai 2026-07-20
ROCm Numa Cache GPU Hardware Inference × Npu Quantization ×
kvcache-ai/ktransformers Python
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
18,506 1,457

KTransformers implements a flexible framework for CPU-GPU heterogeneous LLM inference and fine-tuning, with a focus on MoE models and quantized kernels. It relates to agentic architecture and hardware/software co-design by enabling efficient deployment of large models on consumer hardware through heterogeneous expert placement and CPU-optimized AMX/AVX operations. The repository has a strong upward star trend, currently at 18,506 total stars with 360 stars today. The original paper is published in the ACM Digital Library under DOI 10.1145/3731569.3764843.

Code

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam 2026-07-20
Inference × LLM Quantization × Compiler Low Bit Runtime

PolyQ addresses the problem that existing low-bit quantization for CPU-based LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient to execute. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks with SIMD- and LUT-compatible kernels. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3-bit target, and compiler layout regularization reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets, enabling scalable on-device LLM inference.

PDF