Mohsen Amiri, Ali Beikmohammadi, Sindri Magnússon, Mehdi Hosseinzadeh
8 min
Abstract
This paper addresses the problem of training a reinforcement learning (RL) policy under partial observability by exploiting a privileged, anytime-feasible planner agent available exclusively during training. We formalize this as a Partially Observable Markov Decision Process (POMDP) in which a planner agent with access to an approximate dynamical model and privileged state information guides a learning agent that observes only a lossy projection of the true state. To realize this framework, we introduce an anytime-feasible Model Predictive Control (MPC) algorithm that serves as the planner agent. For the learning agent, we propose Planner-to-Policy Soft Actor-Critic (P2P-SAC), a method that distills the planner agent's privileged knowledge to mitigate partial observability and thereby improve both sample efficiency and final policy performance. We support this framework with rigorous theoretical analysis. Finally, we validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.
Alex: So it's always safe, even if interrupted—like a recipe that works halfway through cooking?
Sam: Exactly. This planner sees the full picture during training and suggests actions on the fly. The key is distilling those suggestions into the robot's own brain, a reactive policy that reacts instantly to what it sees without needing the planner later.
Alex: Distilling—how do they pull that off without the robot just copying blindly and missing its own improvements?
Sam: They train using Soft Actor-Critic, a reinforcement learning method where the robot learns by trial and error to maximize rewards plus some randomness for exploration—like playing a video game, scoring points for progress while trying varied moves to discover better ones. But they add two tweaks: first, imitation in logit space, matching the planner's choices before the final squeeze that turns raw numbers into actions, so learning stays strong even near tricky edges; second, advantage gating, which only copies the planner when it's as good or better, letting the robot override if it finds a smarter move. This combo, called Planner-to-Policy SAC or P2P-SAC, uses a dual buffer to mix real experiences with planner demos.
Alex: Huh—so the imitation avoids flat spots in learning, and the gate keeps it from getting stuck copying bad advice in confusing spots.
Sam: Yes. In tests with a Unitree Go2 quadruped in obstacle mazes, this led to 100% success where plain SAC or PPO failed entirely, as the distillation bridges the info gap without runtime cost. The paper notes this targets the best reactive policy, accepting some limits from partial views.
Alex: That's a clear improvement in reliability. Makes sense why vanilla methods crash. So the gating lets the robot take over when it's better, but keeps the planner's input where needed. Why logit-space specifically for matching the planner?
Sam: Normal imitation compares the final polished actions, but near limits—like max speed or sharp turns—the math signal for tweaks gets tiny and fades out, stalling progress. Instead, they match the raw pre-squeeze numbers, called logits, which stay strong even at extremes; it's like tuning a recipe's ingredients before baking, so changes always register clearly. For a planner action, they reverse-engineer its logit by clipping and inverting the squeeze, ensuring bounded, useful targets.
Alex: Huh, so no vanishing tweaks at boundaries. And the gate—how does it decide when to trust the robot over the planner?
Sam: The gate checks the robot's own value estimate: it computes how much better the planner's suggestion scores compared to the robot's usual choice, using the critic's soft advantage. A sigmoid then weights imitation higher if the planner wins, dialing it down smoothly when the robot improves—keeping guidance in fuzzy spots without blocking better ideas. They follow a three-phase schedule: first a steady high-guidance plateau to build basics, then a slow fade of guidance strength, and finally a mature phase where the robot runs freely but keeps a light ongoing nudge.
Alex: That ties the buffers, schedule, and losses together neatly. Does this setup somehow sidestep the aliasing problem in the learning signals?
Sam: Yes, and the paper proves it mathematically. In the mature phase, the total update splits into the usual robot learning plus a regularizer that pulls the policy toward a weighted average of the planner's raw suggestions across all situations that look the same from the robot's view. Crucially, the extra spread from those lookalikes shows up as a constant term that doesn't change with policy tweaks, so it vanishes from the direction of improvement. This injects privileged know-how without aliasing noise derailing progress.
Alex: So the math averages planner advice over confusable states, but keeps the pull clean and bounded. No wonder it stabilizes where others fail. What stands out most from the tests on that Unitree Go2 hardware?
Sam: In simulations, the approach reached 100% success rates with zero crashes, where standard SAC and PPO failed completely. On the physical Unitree Go2 quadruped, it navigated obstacle-rich setups reliably, staying within safe speed limits and reaching goals. Paths were nearly as efficient as the planner itself.
Alex: That's solid transfer to hardware—no small feat with just position data. But are there built-in limits to what a purely reactive policy like this can handle?
Sam: Yes, the paper is clear on that. Reactive policies, which decide based only on the current view, can't fully close the gap to optimal paths that use full history or beliefs over past states—there's an inherent optimality limit from partial observability. Also, the planner's linear model might not perfectly match the robot's real nonlinear movements, potentially capping guidance quality in edge cases.
Alex: Right, so it excels at reactive avoidance but leaves room for history-aware upgrades. Practically, where does this leave robots or drones in messy real worlds?
Sam: It points toward reliable navigation in unknown 3D spaces using affordable cameras alone, matching human-level obstacle dodging without costly full-state sensors like lidars. The framework—pairing this anytime-feasible MPC planner with the distillation method—shows how to bootstrap robust policies in partial-view settings during training only.
Alex: A meaningful step for standalone robots, grounding the theory in working hardware. Balances strengths with honest limits. Thanks for joining us on ResearchPod.