Offline Reinforcement Learning for Warehouse SLAM Throughput Control

In large-scale warehouse fulfillment centers, Scan/Label/Apply/Manifest (SLAM) systems often face congestion when upstream processing rates exceed downstream capacity. This leads to reactive, rule-based throttling that can disrupt workflows. The authors investigate whether offline reinforcement learning (RL) can learn a more adaptive, data-driven control policy that balances throughput maximization with system stability without requiring risky online experimentation.

The researchers developed an algorithm-agnostic RL framework that treats SLAM throughput control as a Markov Decision Process. To handle the complexities of warehouse operations, they implemented three key design choices:

1. Temporal-aware state representation: The state includes multi-step historical signals of system throughput and upstream activity, allowing the agent to anticipate future arrival rates and understand delayed system responses.
2. Action space abstraction: By aggregating discrete machine settings into time-weighted averages over fixed intervals, the authors reduced a high-dimensional, intractable action space into a compact continuous one.
3. Balanced reward formulation: The reward function normalizes and combines upstream backlog rates and downstream congestion signals to ensure stable gradient updates.

The framework was trained using historical logs from a large-scale warehouse and evaluated using a combination of model-free methods (such as Fitted Q Evaluation) and model-based methods (using Deep Koopman dynamics) to simulate long-horizon policy performance.

Empirical results show that the CQL-based policy consistently outperformed other tested algorithms (BCQ and TD3+BC). By effectively learning the latent dynamics of the warehouse, the CQL policy achieved a 22.97% improvement in system health metrics and a 3.18% reduction in the duration of throttling interventions. This demonstrates that offline RL can successfully optimize complex, delayed-impact industrial processes using existing operational data.