Filtered by: Training × Model × Language × LLM × Clear all

OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters

D. Balamurugan, Thomas W. Bush 2026-07-05
Inference Language × Model × LLM × Quantization Training × GPU Hardware

OmniPilot addresses the challenge of selecting GPU type, tensor-parallel degree, and precision for LLM serving on heterogeneous clusters, where static configurations fail due to fluctuating throughput, launch success, and hardware failures. The method combines a conformally calibrated quantile cost model for eight serving targets with an out-of-distribution (OOD) abstention layer, ranking configurations by an economic utility metric. Across 460 benchmark runs on A100, H100, and H200 hardware, OmniPilot achieves 6.2% MAPE for throughput prediction, 95% top-1 accuracy, and 0.003 mean utility regret, while its abstention layer correctly flags all OOD scenarios. This matters because it enables robust, adaptive LLM deployment on shared clusters, reducing wasted node-hours and improving serving reliability.

PDF

DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

Haotian Xie, Junlin Chen, Mingkai Zheng, Lishan Yang 2026-07-05
Language × Model × LLM × Training × Hardware Runtime

DeadPool addresses the problem of high overhead and long recovery latency in LLM training fault tolerance by proposing a hot-swapping mechanism that replaces failed nodes with spare nodes without terminating the job. The method uses off-critical-path in-memory checkpointing for spatial redundancy and a communicator reconstruction protocol, overlapping checkpointing with computation to achieve zero overhead during error-free execution. Experimental evidence on up to 512 NVIDIA A100 GPUs and LLMs up to 65B parameters shows zero checkpoint overhead and hot-swapping recovery in under 40 seconds. This matters because it simultaneously eliminates failure-free overhead and minimizes recovery cost, enabling resilient large-scale LLM training.

PDF