Dreamer-SAC: Off-Policy Learning in Latent World Models for Sample-Efficient Autonomous Driving

This paper introduces Dreamer-SAC, a model-based reinforcement learning framework designed to address the sample-efficiency bottleneck in autonomous driving. While traditional model-based methods often rely on on-policy optimization within a learned world model, they frequently suffer from model bias when the agent deviates from the distribution of the training data. Dreamer-SAC overcomes this by combining a Recurrent State-Space Model (RSSM) with an off-policy Soft Actor-Critic (SAC) algorithm. By using real-world interaction data to initialize short-horizon latent rollouts, the framework allows the agent to learn from both actual experience and synthetic, model-generated trajectories.

The framework employs a hybrid replay strategy where real transitions are used for one-step temporal-difference learning, while model-generated trajectories utilize n-step target estimation. This dual-target approach allows the agent to exploit long-term reward information within the rollout horizon while maintaining the stability of standard off-policy learning. Furthermore, the model uses a multi-objective reward structure—separating efficiency, lane-keeping, and safety-critical termination signals—to better handle the complex, multi-faceted nature of autonomous driving tasks.

Experiments demonstrate that Dreamer-SAC consistently outperforms representative baselines like DreamerV3, SAC, and PPO in autonomous driving scenarios. A critical finding is the inverted-U relationship between the rollout horizon and policy performance: short-horizon rollouts provide the optimal trade-off between generating useful training signals and avoiding the accumulation of prediction errors inherent in world models. Additionally, the use of n-step targets for predicted experience proved significantly more effective than one-step targets for value learning.