Filtered by: Inference × LLM × Clear all

Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent

Lingyun Yang, Yuxiao Wang, Shenghao Liang, Linfeng Yang 2026-07-19
Inference × LLM × GPU Hardware Agent

Atrex-Bench addresses the problem that existing GPU kernel benchmarks use synthetic or curated workloads, not production traces. The method samples 30 operators and 440 shapes from full-cluster inference traces, weighting each by GPU time and card-hours. Experimental evidence shows the best vanilla model reaches only ~10% of roofline performance, with much apparent correctness from PyTorch fallbacks. This matters because the co-released Atrex-Kernel-Agent (AKA) converts fallbacks into kernels matching hand-tuned baselines, demonstrating a path to production-ready LLM-generated kernels.

PDF

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam 2026-07-19
Inference × LLM × Quantization Compiler Low Bit Runtime

PolyQ addresses the problem that existing low-bit quantization for CPU LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient on CPUs. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses compile-time permutation and clustering to generate SIMD- and LUT-compatible kernels with layout regularization off the runtime path. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3b target and reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient for scalable edge inference.

PDF