FAILUREATLAS: A TAXONOMY OF FAILURE MODES IN MULTI-PROVIDER LLM SERVING INFRASTRUCTURE

Multi-provider LLM gateways—reverse proxies that manage routing, load balancing, and rate limiting across various foundation-model APIs—have become essential infrastructure. However, these systems occupy a unique architectural position that inherits failure modes from both traditional distributed systems and LLM-specific concerns, such as streaming token protocols and stateful conversation management. Despite their criticality, these failure modes have remained undocumented and scattered across disparate issue trackers.

The authors introduce FailureAtlas (FA), a two-axis taxonomy designed to classify these infrastructure failures. The first axis, Layer, categorizes failures by their origin: Network/Transport, Streaming/Protocol, State/Session, Model Behavior, or Governance/Cost. The second axis, Detectability, distinguishes between 'Loud' failures (which trigger standard alerts like 5xx errors) and 'Silent' failures (which return HTTP 200 but corrupt data).

The study's principal finding is that silent failures are the most dangerous. Because they bypass traditional monitoring metrics like latency, error rates, and pod health, they do not trigger incident-response pipelines. Instead, they persist as invisible technical debt, degrading user experience over time. The authors demonstrate that detecting these failures requires semantic-level observability—such as monitoring conversation-state integrity or tool-call payload correctness—rather than relying on infrastructure-level status codes.

The authors populate the taxonomy with five verified catalog entries, including a concurrency race condition that causes history loss and a streaming index collision that corrupts tool calls. By providing mechanistic root-cause analyses and, in three cases, standalone reproduction scripts, the authors aim to provide a shared vocabulary for practitioners to identify and mitigate these complex, domain-specific bugs.