Filtered by: GPU × Clear all

kvcache-ai/ktransformers

kvcache-ai 2026-07-20
ROCm Numa Cache GPU × Hardware Inference Npu Quantization
kvcache-ai/ktransformers Python
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
18,506 1,457

KTransformers implements a flexible framework for CPU-GPU heterogeneous LLM inference and fine-tuning, with a focus on MoE models and quantized kernels. It relates to agentic architecture and hardware/software co-design by enabling efficient deployment of large models on consumer hardware through heterogeneous expert placement and CPU-optimized AMX/AVX operations. The repository has a strong upward star trend, currently at 18,506 total stars with 360 stars today. The original paper is published in the ACM Digital Library under DOI 10.1145/3731569.3764843.

Code

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao 2026-07-20
Attention Inference Training GPU ×

LongStraw addresses the growing gap between inference context lengths and RL post-training, where inference systems handle million-token contexts but post-training typically stays at 256K tokens or below, especially critical for AI agents with long trajectories. The method is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, using GRPO to evaluate shared prompts without autograd, retain only model-specific state, and replay short response branches one at a time to reduce the live training graph. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8, with only 0.21 GB added peak memory per group size increase, and a separate stress test reaches 4.46M positions; on 32 H20 GPUs, it validates the end-to-end execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. This matters because it demonstrates execution capacity for million-token RL post-training under fixed GPU budgets, though the abstract does not disclose complete training correctness due to detached prompt state and incomplete distributed paths.

PDF