Filtered by: Inference × Clear all

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Krish Agarwal, Zhuoming Chen, Yanyuan Qin, Zhenyu Gu 2026-07-23
Inference × Compiler GPU Hardware Model Parallelism Serving Agent

FlashRT addresses the problem of efficiently deploying real-time multimodal applications by automating the optimization of heterogeneous model pipelines. It introduces a chain-of-program paradigm that guides a coding agent to transform reference implementations into an intermediate representation, validate it, and iteratively optimize deployments. On NVIDIA B200 GPUs, FlashRT achieves up to ~70x latency reduction and 2.8x throughput improvement, while on AMD MI355X GPUs it matches peak latency reduction and increases throughput to 3.6x, outperforming expert implementations like vLLM-Omni. This matters because agent-driven optimization enables scalable, high-performance deployment across diverse hardware platforms without requiring hand-crafted implementations.

PDF

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Fabian Waschkowski, Prabod Rathnayaka, Lukas Wesemann 2026-07-23
Attention Inference × Language Model LLM Accelerator GPU Hardware

The paper addresses the problem of maximizing LLM inference throughput on Apple M5 hardware by leveraging its redesigned GPU with dedicated Neural Accelerators. The method, BaseRT, is a native Metal inference runtime that introduces hand-written Metal 4 tensor-core kernels for dense and mixture-of-experts GEMM and flash-attention prefill, routing compute-bound operations through the M5 Neural Accelerators while keeping memory-bound decode on existing kernels. On an M5 Pro across fifteen model configurations from Qwen3, Llama 3.2, and Gemma 4 families, BaseRT achieves up to 6.4× higher prompt-processing throughput than llama.cpp and 3.9× higher than MLX, with decode improvements up to 1.75× and 1.33× respectively. This establishes a new performance ceiling for on-device LLM inference, demonstrating that M5 tensor cores are decisive for prompt processing on Apple Silicon.

PDF

A Flexible Sparsity-Aware FPGA Accelerator with Column-Wise Compression for Efficient CNN Inference

Amirhossein Zarei, Shervin Vakili 2026-07-23
Microarchitecture Simulation Sparsity Accelerator Inference ×

SparHiXcel-v2 addresses the problem of balancing sparsity flexibility and hardware efficiency in CNN acceleration on resource-constrained platforms. The method introduces a column-wise kernel compression technique within a scalable two-dimensional MAC array and a hardware-algorithm co-design framework with ordering optimization and multi-phase structured pruning. Experimental results on VGG16 and ResNet18 show SparHiXcel-v2 achieves over 2.5 TOPS and 210 GOP/s/W for VGG16 on an AMD Kintex UltraScale+ FPGA with modest accuracy loss. This matters because it enables efficient, flexible sparsity-aware CNN inference on cost-effective FPGAs, improving throughput and energy efficiency for edge deployments.

PDF