Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference

As LLM agents become more complex and long-running, they maintain significant state on GPUs, including KV caches, scheduler data, and online adapters. Traditional fault-tolerance mechanisms—such as restarting the entire serving stack or manual application-level logging—are either too slow or too brittle to implement across diverse libraries and frameworks. This paper asks: can we build a transparent, high-performance fault-tolerance substrate that operates below the framework level, directly on the GPU, to enable rapid recovery without host-side bottlenecks?

Concordia introduces a persistent-kernel runtime that remains resident on the GPU for the duration of an LLM session. By interposing on GPU module loading, the system uses PTX and SASS instrumentation to insert checkpoint and pause hooks directly into the binaries executed by the framework.

Key technical components include:

• Persistent Executor: A lock-free ring buffer allows the host to submit tasks (compute, checkpoint, or recovery) to a resident GPU worker, avoiding the overhead of repeated kernel launches.
• JIT-Compiled Handlers: Concordia dynamically compiles specialized delta-checkpoint handlers for different memory regions (e.g., PagedAttention KV blocks or LoRA adapters) and hot-swaps them into the persistent kernel's operator table.
• Append-Only Logging: Rather than full snapshots, Concordia logs sparse state changes to CPU-visible CXL memory or host DRAM, mimicking the write-ahead logging found in databases.

Concordia effectively decouples fault tolerance from the host CPU. By performing dirty-page detection and diffing on the GPU at HBM bandwidth, the system achieves up to 219x faster delta-checkpointing compared to host-side page scanning. Because the persistent kernel handles checkpoint triggers and recovery logic, the system can restore service in approximately 1.5 seconds in a two-GPU prototype, significantly faster than restarting the NCCL communicator and reloading model weights.