Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

The Problem with Conventional Scaling

Traditional scaling laws, such as those established by Kaplan et al. and Hoffmann et al., treat model architecture and systems implementation as disconnected stages. Researchers first determine the optimal model size and token count based on a fixed compute budget (Model FLOPs) and subsequently optimize the hardware execution. For sparse Mixture-of-Experts (MoE) models, this approach is problematic because the sparsity factor—the fraction of parameters inactive per token—significantly impacts communication overhead, memory footprint, and hardware utilization (MFU). When using a pure Model FLOPs budget, the loss function often suggests that sparser models are always better, pushing the optimal sparsity to the theoretical boundary of the data support, which is often infeasible or inefficient on actual hardware clusters.

The MOSAIC Approach

MOSAIC (Model Optimization via Systems-Aware Training Co-design) shifts the paradigm by folding systems-level constraints directly into the scaling law optimization. Instead of optimizing for abstract Model FLOPs, MOSAIC optimizes for 'deliverable' Model FLOPs—the actual computation a specific cluster can perform within a fixed wall-clock time window. By coupling a predictive scaling law with a calibrated performance model that accounts for expert parallelism, communication costs, and memory constraints, MOSAIC treats the model architecture, token budget, and parallel execution layout as a single, unified optimization problem.

This work demonstrates that the 'compute-optimal' architecture is not necessarily the 'cluster-optimal' one. By accounting for the realities of distributed training—such as kernel-launch overheads, all-to-all communication, and pipeline bubbles—MOSAIC identifies interior optimal sparsity configurations that yield lower loss than the boundary-seeking configurations suggested by traditional methods. This unified co-design approach is essential for scaling frontier language models, where the gap between theoretical compute and realized hardware performance continues to widen.