A Model-Driven Approach for Developing Families of Reinforcement Learning Environments

Research Context and Motivation

Reinforcement learning (RL) agents often struggle to generalize when trained in a single, static environment. To achieve robust performance, agents typically require exposure to a family of diverse but structurally similar environments. Currently, creating these environment families is a labor-intensive, manual process that does not scale well to the requirements of modern RL applications. This paper addresses this bottleneck by applying model-driven engineering (MDE) principles to automate the generation of these training environments.

A Model-Driven Approach

The authors propose a framework where the environment is treated as a model conforming to a domain-specific metamodel. The process begins with an RL expert defining an initial, challenging environment. A hybrid genetic algorithm (GA) then performs a global search to generate a population of environment variants. These variants are created by applying mutation operators—defined as model transformations—to the initial environment.

To ensure the generated environments are valid and useful, the framework incorporates:

• Constraint Validation: Graph queries ensure that every generated environment adheres to structural requirements (e.g., maintaining specific start or goal locations).
• Heuristic Search: A local search method (such as simulated annealing) selects the most effective subset of environments to form a curriculum.
• Automation: The entire pipeline is operationalized using the Viatra model transformation framework, allowing for the automated generation of both the environment models and the corresponding code for the RL framework.

Evaluation and Impact

The authors demonstrate their approach using a wildfire mitigation scenario, where an agent must learn to navigate a grid world with varying densities of hazards. The results show that the automated approach produces environment families that facilitate effective curriculum learning, where the agent progresses from simpler to more complex tasks. By lifting structural properties to the model level, the approach minimizes human intervention while enabling the creation of large, diverse training sets that would be impractical to design manually.