FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

FlashRT addresses the problem of efficiently deploying real-time multimodal applications by automating the optimization of heterogeneous model pipelines. It introduces a chain-of-program paradigm that guides a coding agent to transform reference implementations into an intermediate representation, validate it, and iteratively optimize deployments. On NVIDIA B200 GPUs, FlashRT achieves up to ~70x latency reduction and 2.8x throughput improvement, while on AMD MI355X GPUs it matches peak latency reduction and increases throughput to 3.6x, outperforming expert implementations like vLLM-Omni. This matters because agent-driven optimization enables scalable, high-performance deployment across diverse hardware platforms without requiring hand-crafted implementations.