Daily | Yixun Hong

kvcache-ai/ktransformers

kvcache-ai 2026-07-20

ROCm Numa Cache GPU Hardware Inference Npu Quantization

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

KTransformers implements a flexible framework for CPU-GPU heterogeneous LLM inference and fine-tuning, with a focus on MoE models and quantized kernels. It relates to agentic architecture and hardware/software co-design by enabling efficient deployment of large models on consumer hardware through heterogeneous expert placement and CPU-optimized AMX/AVX operations. The repository has a strong upward star trend, currently at 18,506 total stars with 360 stars today. The original paper is published in the ACM Digital Library under DOI 10.1145/3731569.3764843.

Code

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao 2026-07-20

Attention Inference Training GPU

LongStraw addresses the growing gap between inference context lengths and RL post-training, where inference systems handle million-token contexts but post-training typically stays at 256K tokens or below, especially critical for AI agents with long trajectories. The method is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, using GRPO to evaluate shared prompts without autograd, retain only model-specific state, and replay short response branches one at a time to reduce the live training graph. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8, with only 0.21 GB added peak memory per group size increase, and a separate stress test reaches 4.46M positions; on 32 H20 GPUs, it validates the end-to-end execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. This matters because it demonstrates execution capacity for million-token RL post-training under fixed GPU budgets, though the abstract does not disclose complete training correctness due to detached prompt state and incomplete distributed paths.

PDF

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam 2026-07-20

Inference LLM Quantization Compiler Low Bit Runtime

PolyQ addresses the problem that existing low-bit quantization for CPU-based LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient to execute. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks with SIMD- and LUT-compatible kernels. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3-bit target, and compiler layout regularization reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets, enabling scalable on-device LLM inference.

PDF

ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

Hyunwoo Oh, Suyeon Jang, Hanning Chen, Sanggeon Yun 2026-07-20

Gem5 Microarchitecture Simulation Inference

The problem is that low-bit GEMM is central to efficient ML inference, but very-low-bit execution is poorly suited to conventional CPUs, and the fragmented design space of precision regimes makes lightweight CPU support selection a first-class challenge. ExaGEMM proposes a workload-aware codesign framework for CPU-native low-bit GEMM using register-resident LUT execution, requiring only an in-register select/feed mechanism as new hardware. Experimental evidence shows ExaGEMM improves latency by 13.29x over software-only baselines across representative ML models and CPU targets, while pruning the candidate space by 99.2% before simulation. This matters because it enables practical, efficient CPU-driven ML inference for mixed-precision LLM workloads, demonstrating that workload-aware frontier selection is critical for low-bit GEMM support.

PDF

NIFA: Nonlinear IMC enhanced FPGA for efficient ML inference

Jiajun Hu, Ruthwik Reddy Sunketa, Lei Zhao, Archit Gajjar 2026-07-20

Design Space Exploration Agentic Architecture Inference

The problem is that conventional ReRAM-based IMC blocks in FPGAs only support static-weight VMM, limiting efficiency gains for Transformer models that require nonlinear and dynamic matrix-matrix multiplication (DIMM) operations, while ADCs consume over 70% of IMC block area and power. The method proposes NIFA, a novel FPGA architecture integrating an ADC-free IMC block that replaces ADCs with analog content-addressable memories (ACAMs) to natively perform nonlinear operations, combined with FPGA-aware design-space exploration and an efficient mapping for DIMM. Experimental evidence shows up to 40x and 1.9x higher energy efficiency and 4.1x and 2.5x higher area efficiency on CNN and Transformer-based benchmarks, respectively. This matters because it significantly improves FPGA DL inference efficiency and sustains robust gains on Transformer-based workloads, advancing domain-specialized FPGA design for modern ML models.

PDF

RTL-Sequencer: Towards Scalable RTL Timing Prediction with the Sequence-based Paradigm

Ziyan Guo, Wenji Fang, Wenkai Li, Yuchao Wu 2026-07-20

Design Architecture

The problem is that existing graph-based RTL timing prediction methods suffer from limited receptive fields, high complexity, and a lack of signal directionality. RTL-Sequencer introduces a sequence-based paradigm that linearizes logic cones via breadth-first traversal and applies modern linear sequence models with four synergistic techniques. Experimental results show significant improvements over state-of-the-art baselines, demonstrating the method's effectiveness. This matters because it enables more scalable and accurate early-stage timing optimization in design automation.

PDF

CoG-Guided Weight Correction for Fault-Tolerant Deep Neural Networks

Bahram Parchekani, Samira Nazari, Ali Azarpeyvand, Mohammad Hasan Ahmadilivani 2026-07-20

Neural Network Hardware

The paper addresses the vulnerability of DNN weights to hardware faults in safety-critical systems. It introduces a Center of Gravity (CoG)-guided method that corrects faulty weights using spatial, distance-aware rules without retraining or architecture changes. Fault injection experiments on LSTM networks (StageNet, MTFNet) show up to 230x and 6.41x fault tolerance improvements at a BER of 10^{-3}, and on CNNs (ResNet-18, VGG-16) up to 49.55x and 20.79x improvements, with negligible accuracy loss. This matters because it is the first application of CoG to weight tensors, offering a practical, retraining-free approach to enhance DNN reliability in safety-critical applications.

PDF

Yarrow: Reconciling Effects Handlers and Region-Based Memory Management

Anders Alnor Mathiasen, Amin Timany, Lars Birkedal 2026-07-20

Runtime Data

The problem is that algebraic effects with multi-shot handlers break the stack discipline and region-exit invariants required by region-based memory management. The method is a new language, Yarrow, with a program logic called Yarrow Logic (YL) for safe, modular reasoning about regions under one-shot and multi-shot effect handlers. Experimental evidence shows YL proves correctness of case studies including checkpointing, asynchronous computation, and a LIFO data structure, all using region-allocated memory to avoid garbage collection. This matters because it enables efficient, region-managed memory in languages with algebraic effects, eliminating reliance on garbage-collected heap memory.

PDF

A Modular Framework for Stack-Heap and Value Abstractions (Extended Version)

Giacomo Boldini, Luca Negrini, Luca Olivieri, Pietro Ferrara 2026-07-20

Data Runtime

This paper addresses the problem of designing a static analysis framework that can accurately model stack and heap memory across diverse programming languages. The method proposes a modular, parametric framework based on Abstract Interpretation that separates value and memory analyses into interacting abstract domains via memory identifiers. The abstract does not disclose experimental results, but the framework is formalized to support sound approximation of memory models for languages like C, C++, Java, and Python. This matters because it enables detection of critical memory errors (e.g., buffer overflows, use-after-free) before execution, while allowing independent selection and combination of different analyses.

PDF

exa-PD: A scalable high-performance workflow for multi-element phase diagram construction

Zhuo Ye, Feng Zhang, Maxim Moraru, Weiyi Xia 2026-07-20

Cond Mat Mtrl Sci Physics Comp Ph

The problem is the computational bottleneck in constructing multi-element phase diagrams (PDs) due to the need for extensive free-energy sampling. Exa-PD introduces a highly parallelizable workflow using Parsl to coordinate large ensembles of LAMMPS-based molecular dynamics and Monte Carlo tasks across a fine temperature-composition mesh. Experimental evidence demonstrates strong scalability and massive parallelization, enabling efficient free-energy calculations for liquid and solid phases. This matters because it integrates with PyCalphad for CALPHAD modeling, making high-throughput multi-element PD construction feasible for materials science.

PDF