DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

DeadPool addresses the problem of high overhead and long recovery latency in LLM training fault tolerance by proposing a hot-swapping mechanism that replaces failed nodes with spare nodes without terminating the job. The method uses off-critical-path in-memory checkpointing for spatial redundancy and a communicator reconstruction protocol, overlapping checkpointing with computation to achieve zero overhead during error-free execution. Experimental evidence on up to 512 NVIDIA A100 GPUs and LLMs up to 65B parameters shows zero checkpoint overhead and hot-swapping recovery in under 40 seconds. This matters because it simultaneously eliminates failure-free overhead and minimizes recovery cost, enabling resilient large-scale LLM training.