Onno Eberhard, Thibaut Cuvelier, Michal Valko, Bruno De Backer
5 min
Middle-mile logistics involves routing parcels through a network of hubs using trucks with finite capacity. Because trucks have limited space, routing decisions are interdependent; assigning a parcel to one truck may prevent another parcel from using it. Traditional methods, such as mixed-integer programming, struggle to scale to the large, real-world instances common in modern logistics. This paper investigates whether reinforcement learning (RL) can provide a scalable alternative to these traditional, computationally expensive methods.
The authors reframe the middle-mile problem as a multi-object goal-conditioned Markov decision process (MDP). They represent the logistics network as a time-expanded graph, where nodes represent hub-time locations and edges represent truck connections. To handle the large state space, the authors use graph neural networks (GNNs) to process small, local feature graphs extracted from the environment. They employ a custom proximal policy optimization (PPO) algorithm to learn routing policies, using masking to ensure that only valid truck assignments are considered at each step.
The researchers demonstrate that their GNN-based RL approach can learn to route parcels in simulated logistics networks. However, experimental results show that the current RL implementation is outperformed by simpler greedy heuristics and supervised learning baselines. The authors argue that while these results are modest, the RL framework is more flexible than static heuristics, as it can be adapted to more complex cost functions and stochastic environments where traditional methods or supervised learning (which requires optimal labels) might fail.
This work represents an early attempt to apply modern deep reinforcement learning to the complex, combinatorial challenge of middle-mile logistics. By open-sourcing their environment, the authors aim to bridge the gap between the RL and logistics research communities, encouraging the development of more scalable, learning-based solutions for real-world supply chain management.
Middle-mile logistics describes the problem of routing parcels through a network of hubs linked by trucks with finite capacity. We rephrase this as a multi-object goal-conditioned MDP. Our method combines graph neural networks with model-free RL, extracting small feature graphs from the environment state.
Alex: That's the goal. And to process the web of hubs and connections, they use a type of neural network specifically designed for this kind of data. Think of a regular neural network as something that reads a list. A graph neural network, by contrast, reads a map—it understands that hub A connects to hub B, and hub B connects to hubs C and D. It can pass information along those connections, so each hub's understanding is shaped by what its neighbors are doing.
Sam: That makes sense for understanding the local area. But if the AI only sees a small patch of the network, how does it avoid making a locally smart choice that's globally terrible? Like taking a shortcut that leads to a dead end three hubs later?
Alex: That's a real concern, and the paper addresses it directly. Even within the local view, they include extra information—specifically, estimates of how far the parcel still needs to travel to reach its goal. It's a bit like a hiker who can only see the next valley but still has a compass pointing toward the mountain peak. The local decisions are informed by a sense of the broader direction.
Sam: Does that actually hold up in practice?
Alex: The results suggest it does. When tested against a simpler, linear baseline method, the graph neural network approach delivers a clear improvement in the fraction of parcels successfully reaching their destinations. The model demonstrably learns the routing task.
Sam: You mentioned they use something called Proximal Policy Optimization to update the AI's strategy during training. What's that about?
Alex: Think about learning to ride a bike. If someone shoves you hard to correct your balance, you fall off entirely. Small, careful corrections work much better. Proximal Policy Optimization is essentially that principle applied to AI training—it's a rule that says the strategy can only change by a small, controlled amount with each update. Too big a change and the whole learning process becomes unstable.
Sam: So it's a guardrail that keeps the learning process from overcorrecting.
Alex: Exactly. And alongside that, the authors use a technique to measure not just whether a delivery succeeded, but how much better a specific routing choice was compared to what the AI would have done on average. It helps the AI focus its learning on the decisions that actually made a meaningful difference, rather than just getting credit for easy wins.
Sam: And the paper makes the whole simulation environment open-source?
Alex: It does. That's a notable aspect of this work. By releasing the code, they give other researchers a shared testing ground—the same complex, time-sensitive network—so that different routing strategies can be compared fairly. It's an invitation for the broader research community to build on this foundation.
Sam: So the contribution here is really two things: a working approach to this problem, and a platform for others to improve on it.
Alex: That's a fair summary. The core insight is that a problem which looks impossibly large at the global scale becomes tractable when you decompose it into local, learned decisions. The results are promising, the authors are candid about the open challenges that remain, and the open environment means this is a starting point rather than a final answer. Thanks for listening to ResearchPod.