Filtered by: GPU × Serving × Clear all

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Krish Agarwal, Zhuoming Chen, Yanyuan Qin, Zhenyu Gu 2026-07-22
Inference Compiler GPU × Hardware Model Parallelism Serving × Agent

FlashRT addresses the problem of manually optimizing real-time multimodal application deployments across heterogeneous models and hardware. It introduces an agent harness that uses a chain-of-program paradigm to transform simple reference implementations into optimized multi-GPU deployments via iterative IR construction, validation, and measurement-gated optimization. On NVIDIA B200 GPUs, FlashRT achieves up to ~70x latency reduction and 2.8x throughput improvement, while on AMD MI355X GPUs it matches peak latency reduction and increases throughput improvement to 3.6x, outperforming expert implementations like vLLM-Omni by 65% in response latency. This matters because agent-driven optimization enables scalable, high-performance deployment across diverse hardware platforms without requiring hand-crafted expert tuning.

PDF

ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels

Bingyang Wu, Chao Jin, Zili Zhang, Xinming Wei 2026-07-22
Attention Fp8 GPU × Serving × LLM

ExpertPlex addresses the problem of inefficient resource utilization in serving large Mixture-of-Experts (MoE) LLMs, where existing prefill-decode disaggregation wastes resources due to coarse allocation and colocation suffers from phase interference and poor load tracking. The method shares massive MoE experts across phases while disaggregating attention modules, and introduces adaptive persistent kernels, attention-initiated MoE communication, and a tile-to-cluster model to optimize execution. Experiments serving MiniMax-M2.7 and GLM-5.1-FP8 show ExpertPlex improves goodput by up to 2.01× over instance-level disaggregation and 1.66× over colocation. This matters because it enables efficient, high-throughput serving of increasingly large MoE models without overprovisioning or phase interference.

PDF