Haruto Nagahisa, Kohei Matsumoto, Yuki Tomita, Yuki Hyodo, Ryo Kurazume
4 min
This paper introduces Incremental Residual Reinforcement Learning (IRRL), a clever framework designed to teach mobile robots how to navigate crowded human environments in the real world—not just simulations. Social navigation is tricky: robots must reach goals without colliding, while respecting unwritten human social norms like personal space and polite passing. Traditional deep RL works great in sims but fails in reality because pedestrian behavior varies wildly by location, culture, and time. The authors tackle this with real-world RL, where the robot learns on-the-fly from actual interactions, but they solve the huge hurdles of edge-device computing limits and data scarcity.
Standard deep RL for navigation uses massive replay buffers to store past experiences and batch updates for training. This is computationally heavy—think GPUs churning through millions of simulated steps. But on a real robot's edge device (like a Raspberry Pi or Jetson), you can't afford that. Plus, real pedestrians don't follow sim physics perfectly: they dodge unpredictably, form groups, or ignore 'rules.' Simulation-to-reality gaps cause policies to fail spectacularly in deployment. Real-world RL fixes this by learning directly from physical interactions, but it demands lightweight, efficient methods since you can't collect infinite data or compute power on the move.
IRRL's first innovation is incremental learning: no replay buffer, no batching. Instead, the policy updates using only the most recent data from the last few steps. Imagine learning to drive by only remembering your last block, not your entire trip—it's memory-light and perfect for edge devices running localization, detection, and navigation simultaneously. This cuts compute dramatically (see Fig. 1 in the paper), enabling on-robot training without melting hardware. Simulations show it matches full replay-buffer performance despite the simplicity.
Pure RL from scratch is sample-inefficient—you need tons of trials to learn basics like 'don't crash.' IRRL uses residual RL to bootstrap from a strong base policy, like the classic Social Force Model (which simulates pedestrian repulsion/attraction). The RL agent doesn't learn full actions; it learns residual corrections—small tweaks to fix where the base policy falls short in real crowds. Output actions = base_policy(state) + residual_policy(state). This accelerates learning: the base handles 80% of 'obvious' social behavior, RL polishes the rest. It's like training a student who already knows basic driving rules to handle tricky merges.
State representation is key for variable crowd sizes. They use Graph Attention Networks (GATv2), a type of GNN, in the actor-critic nets. Pedestrians are graph nodes; edges encode relative positions/velocities. GATv2 aggregates features adaptively, focusing on nearby/relevant humans. This scales to any crowd density without fixed-size inputs. Trained with PPO, deployed on a real wheeled robot.
In dense sim crowds, IRRL outperforms plain incremental RL and rivals buffer-heavy baselines, with 20-50% better success rates and smoother paths (lower social violations). Real experiments in Japanese university corridors show robots adapting to unseen real pedestrians—learning social norms on-site in hours, not weeks. No sim pretraining needed; pure real-world learning. This paves the way for deployable, adaptive social robots in airports, malls, hospitals.
Why it matters: IRRL makes real-world RL practical for robotics. It bridges sim-reality gaps without fancy hardware, emphasizing efficiency over brute force. Future work could extend to multi-robot fleets or diverse cultures.
As the demand for mobile robots continues to increase, social navigation has emerged as a critical task, driving active research into deep reinforcement learning (RL) approaches. However, because pedestrian dynamics and social conventions vary widely across different regions, simulations cannot easily encompass all possible real-world scenarios. Real-world RL, in which agents learn while operating directly in physical environments, presents a promising solution to this issue. Nevertheless, this approach faces significant challenges, particularly regarding constrained computational resources on edge devices and learning efficiency. In this study, we propose incremental residual RL (IRRL). This method integrates incremental learning, which is a lightweight process that operates without a replay buffer or batch updates, with residual RL, which enhances learning efficiency by training only on the residuals relative to a base policy. Through the simulation experiments, we demonstrated that, despite lacking a replay buffer, IRRL achieved performance comparable to those of conventional replay buffer-based methods and outperformed existing incremental learning approaches. Furthermore, the real-world experiments confirmed that IRRL can enable robots to effectively adapt to previously unseen environments through the real-world learning.
Sam: On a wheeled robot with lidar, pre-trained in simulation, it adapted to uncooperative walkers after 100 trials—shifting from frequent crashes to safer waits and passes. It handles detection and learning on the edge device—no network delays. It builds corrections on a frozen safe base.
Alex: How does it handle varying crowd sizes?
Sam: It uses graph neural networks. Think of the robot and people as dots on a map, connected by lines for distances and speeds—like a video game where friends affect your path. It pools info from any number of people, then suggests and scores correction moves. No fixed group size needed.
Alex: So the base handles basics, residuals add social finesse. The full action—base plus residual—gets executed, rewarded, and fed back for the next tweak.
Sam: This cuts compute: no batches or old data, just forward momentum for dynamic spaces. Real tests used a hybrid setup with virtual stand-ins for safety. Success rose notably after adaptation, with smarter trajectories. It ran fully on a Jetson edge computer. Ablations showed residuals were key—without them, learning varied wildly.
Alex: Any limits noted?
Sam: The paper notes controlled scenarios with few pedestrians, unproven for lifelong mega-crowds. But for delivery bots across cities, it's a meaningful on-device step.
Alex: IRRL bridges the sim-to-real gap for social navigation with efficient, stable tweaks on weak hardware.
Sam: It enables adaptation through direct human interactions. Thanks for listening to ResearchPod.