Cost-Optimal Foundation Model Deployment Portfolio for Transportation Management

Transportation management centers (TMCs) increasingly rely on foundation models (FMs) for tasks like incident detection and traffic state estimation. However, agencies face a complex portfolio problem: they must select from a mix of closed-source APIs, open-source hosted models, and on-premise hardware while balancing strict requirements for quality, latency, safety, and shared GPU capacity. This paper introduces the Foundation Model Deployment Portfolio (FMDP) framework to solve this resource allocation challenge.

The authors formulate FMDP as a mixed-integer program that minimizes the total cost of ownership (TCO) subject to per-function constraints. Because the problem is NP-hard, they propose a two-phase greedy heuristic: the first phase assigns functions to their cheapest feasible model-mode pair, and the second phase iteratively reassigns functions to API fallbacks if the GPU capacity is exceeded. They validate this approach using a case study of five representative TMC functions, comparing the optimized portfolio against various baseline strategies.

The FMDP framework identifies a mixed-deployment strategy that satisfies all operational constraints at a cost of $34/month, which is 97% cheaper than the most cost-effective all-closed-API baseline. The analysis reveals that for most TMC functions, hosted open-source APIs are currently more cost-effective than maintaining on-premise GPU infrastructure. On-premise hardware only becomes the optimal choice when query volumes are high enough to amortize the fixed costs of GPU ownership—specifically, beyond 309 vision queries per hour—or if cloud API pricing increases significantly.

As agencies transition to AI-driven traffic management, they lack formal frameworks to decide between cloud-based and on-premise infrastructure. This research provides a rigorous, scalable method for TMCs to manage their AI budgets while ensuring that safety-critical functions (like wrong-way driver detection) meet necessary quality and latency standards. It highlights the economic trade-offs between data sovereignty, hardware investment, and the flexibility of cloud-based model consumption.