Riccardo De Monte, Matteo Cederle, Gian Antonio Susto
10 min
Modern deep reinforcement learning (RL) typically relies on batch-based training, which utilizes replay buffers, target networks, and mini-batch updates to ensure sample efficiency and training stability. However, these components are computationally demanding and memory-intensive, making them ill-suited for real-time on-device learning or resource-constrained robotic systems. The authors investigate streaming deep reinforcement learning, a paradigm that performs updates purely online without a buffer, to bridge the gap between initial simulated training and real-world deployment (the Sim2Real gap).
The authors propose two algorithms, S2AC and SDAC, to enable online adaptation for continuous control tasks. These methods incorporate critical refinements such as state normalization, reward scaling, and LayerNorm to handle training instabilities. A key methodological contribution is the introduction of a time-varying entropy coefficient for S2AC to prevent policy collapse caused by reward normalization. For SDAC, the authors introduce target noise injection to stabilize Q-value estimation in a streaming context. Both algorithms are designed to be architecture-compatible with widely used batch RL methods like SAC and TD3.
A central focus of the paper is the transition from a pre-trained batch policy to a streaming adaptation phase. The authors discover that a naive transition—simply switching the training loop—causes a significant drop in performance. By analyzing the geometry of the learned weights, they demonstrate that standard batch optimizers (e.g., Adam) and streaming optimizers (e.g., ObGD) induce different solution landscapes. To mitigate this, they propose replacing Adam with an equivalent stochastic gradient descent variant (SGDC) during batch pre-training, which aligns the solution geometry and enables a smooth, stable transition to streaming finetuning.
This work provides a practical pathway for deploying deep RL on edge devices. By enabling agents to continue learning on-board after being pre-trained in simulation, this approach addresses the Sim2Real gap and allows robotic systems to adapt to dynamic environments without needing the heavy compute required by batch methods. The findings underscore that architectural and methodological choices in batch RL, such as normalization techniques, can be effectively ported to streaming setups to improve performance.
State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of-the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious per-environment hyperparameter tuning. We further investigate the batch-to-streaming transition, showing that a naive transition does not guarantee preservation of pre-trained policy performance, and propose a principled approach to address this challenge.
Sam: So the novelty isn’t simply using fewer stored experiences. It’s keeping a structure that a pretrained controller can actually transfer into.
Alex: That is the intended contribution. They introduce Streaming Soft Actor-Critic, or S2AC, and Streaming Deterministic Actor-Critic, or SDAC. Both use a critic: a network estimating future reward for a state and action. That matches the kind of value estimate used by their batch counterparts.
Sam: What keeps those value estimates from becoming unstable when each update sees only one experience?
Alex: Several safeguards work together. They normalize observations and scale rewards using statistics tracked online. They also normalize internal network activations and start with sparse network connections. The critic uses eligibility traces, decaying gradient memories that help assign credit to earlier interactions.
Sam: The update size seems especially important here. One unusually large error could move the whole controller in the wrong direction.
Alex: The critic uses Overshooting-bounded Gradient Descent, an optimizer that limits potentially oversized updates. The actor—the network choosing actions—still uses Adam, a standard adaptive optimizer. So this is not a blanket rejection of Adam; the optimizer choice differs between the networks.
Sam: What distinguishes the two new algorithms beyond whether their actions are random or deterministic?
Alex: The stochastic method balances reward against policy entropy, meaning how spread out its action choices are. Scaling rewards changes that balance unless the entropy weight changes too. S2AC scales both together, preserving their relative weighting as reward statistics evolve.
Sam: And for the deterministic method, how do they stop the critic from favoring a very narrow, misleading peak?
Alex: SDAC adds Gaussian noise to the action used to calculate its learning target. That encourages smoother value estimates around an action. In the reported ablation, removing this target noise prevented SDAC from learning. S2AC still performed reasonably without its entropy-scaling correction, so those changes had different importance.
Sam: How broad is the evidence that these algorithms work before we even consider transferring a trained policy?
Alex: They train streaming agents for twenty million environment steps on MuJoCo Gym and DeepMind Control Suite tasks. Stream Actor-Critic is the primary comparison. The experiments use ten random seeds, and the authors report confidence intervals around performance.
Sam: That’s a substantial interaction budget. Does competitive performance mean either new method consistently beats the existing streaming baseline?
Alex: No. Performance is comparable overall, and the strongest algorithm varies by environment. There’s also a tuning qualification: S2AC avoids environment-specific optimizer, learning-rate, and discount-factor tuning. But its entropy temperature still differs across environments. SDAC uses no environment-specific hyperparameters in those from-scratch experiments.
Sam: Let’s turn to the transfer result. If the networks fit together, why can a pretrained agent still lose performance?
Alex: First, they improve the batch starting point using the streaming-style architecture and data normalization. Those variants show gains across many tasks, especially for TD3. But directly switching from normalized TD3 to SDAC can still cause a severe performance drop.
Sam: So architectural compatibility gets the weights across, but doesn’t ensure the new learning rule treats those weights gently. What explanation do the authors test?
Alex: They hypothesize that the critic optimizer shapes the learned solution differently. Batch TD3 uses Adam, while streaming SDAC uses the overshooting-bounded optimizer. The critic weights have substantially different norms under those optimizers. That supports their hypothesis, but doesn’t establish the full mechanism.
Sam: How do they prepare the batch-trained critic for the change?
Alex: They replace its Adam optimizer during pretraining with Stochastic Gradient Descent with Clipping. Clipping limits the size of gradient updates. They show a mathematical connection to the streaming optimizer: under specified single-sample conditions, the update rules coincide.
Sam: That sounds like aligning the learning rules before deployment, rather than repairing the controller after it has already deteriorated. What does their concrete adaptation test look like?
Alex: Take their dog-walking simulation. They reduce joint stiffness, actuator gains, and the available control range to represent changed dynamics. They then transfer a controller pretrained with normalized TD3 and the clipped-gradient critic optimizer into SDAC. Two walker variants test symmetric and asymmetric dynamics changes.
Sam: These are stand-ins for simulation-to-reality differences, not tests on physical robots. What happens during adaptation?
Alex: Finetuning lasts five million steps, compared with the longer from-scratch streaming runs. Across pretrained checkpoints and finetuning seeds, they conduct eighteen experiments per environment. Pretraining accelerates learning and often produces better final performance than SDAC trained from scratch.
Sam: Does that make the switch reliable, or are there still important exceptions?
Alex: There are clear exceptions. SDAC finetuning is ineffective on the quadruped-running task, according to the conclusion. Using clipped-gradient critic pretraining also reduces sample efficiency in some environments. And transferring from SAC to S2AC remains unresolved: both pretraining reliability and entropy selection cause problems.
Sam: The successful pipeline also includes more than an optimizer replacement. What should a researcher avoid overlooking?
Alex: A critic-only warm-up freezes the policy for five thousand steps. The finetuning setup also lowers the actor learning rate and changes noise settings. Ablations suggest both warm-up and explicit exploration noise help stability. I’d read this as evidence for the combined pipeline, not an isolated optimizer cure.
Sam: Who should read the full document, and where would you send them first?
Alex: Researchers working on resource-limited adaptation should start with “Towards Batch-to-Streaming Deep RL,” then Appendix K’s finetuning settings and ablations. Those sections show the failed direct transition, the proposed preparation, and the remaining exceptions. Algorithm designers should also inspect the target-noise and entropy-scaling comparisons.
Sam: And for listeners outside that deployment problem, what is the one line to remember?
Alex: Matching network architecture is not enough to make a learning-mode switch safe. This study offers a promising preparation strategy, but not guaranteed performance preservation or demonstrated physical-robot deployment.
Sam: Thanks for listening.