Filtered by: Inference × Clear all

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Krish Agarwal, Zhuoming Chen, Yanyuan Qin, Zhenyu Gu 2026-07-22
Inference × Compiler GPU Hardware Model Parallelism Serving Agent

FlashRT addresses the problem of manually optimizing real-time multimodal application deployments across heterogeneous models and hardware. It introduces an agent harness that uses a chain-of-program paradigm to transform simple reference implementations into optimized multi-GPU deployments via iterative IR construction, validation, and measurement-gated optimization. On NVIDIA B200 GPUs, FlashRT achieves up to ~70x latency reduction and 2.8x throughput improvement, while on AMD MI355X GPUs it matches peak latency reduction and increases throughput improvement to 3.6x, outperforming expert implementations like vLLM-Omni by 65% in response latency. This matters because agent-driven optimization enables scalable, high-performance deployment across diverse hardware platforms without requiring hand-crafted expert tuning.

PDF

A Flexible Sparsity-Aware FPGA Accelerator with Column-Wise Compression for Efficient CNN Inference

Amirhossein Zarei, Shervin Vakili 2026-07-22
Microarchitecture Simulation Sparsity Accelerator Inference ×

Problem: Efficient CNN inference on resource-constrained FPGAs is challenged by the irregularity of sparsity patterns, where unstructured sparsity causes hardware inefficiencies and structured sparsity sacrifices flexibility. Method: SparHiXcel-v2 introduces a scalable two-dimensional MAC array with column-wise kernel compression for irregular sparsity, plus a hardware-algorithm co-design framework with ordering optimization and multi-phase structured pruning and revival. Finding: On a cost-effective AMD Kintex UltraScale+ FPGA, SparHiXcel-v2 achieves over 2.5 TOPS and 210 GOP/s/W for VGG16, and over 1.1 TOPS and 72 GOP/s/W for ResNet18 in structured sparsity mode with modest accuracy loss. Why it matters: This work provides a flexible, energy-efficient FPGA accelerator that balances sparsity flexibility and hardware efficiency, enabling practical CNN deployment on resource-constrained platforms.

PDF