Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

This study investigates how to optimize Large Language Models (LLMs) for generating Business Process Model and Notation (BPMN) models. While supervised fine-tuning (SFT) allows models to mimic training data, it creates a performance ceiling. The authors explore whether Reinforcement Learning (RL) using Group Sequence Policy Optimization (GSPO) can surpass this ceiling by optimizing against a multi-dimensional reward function. They evaluate 48 configurations across two LLM families (Llama 3.1 8B and Qwen 2.5 14B), using 38 automated metrics that measure syntactic, pragmatic, and semantic quality.

• RL Improves Quality and Consistency: GSPO significantly enhances pragmatic and syntactic quality compared to SFT-only baselines. Crucially, RL reduces output variability by more than sixfold, leading to more reliable and consistent process models.
• Equal Weighting is Superior: Contrary to the expectation that emphasizing a specific quality dimension (e.g., syntax) would improve it, equal reward weighting consistently outperforms targeted weighting. Targeted weighting often causes the model to collapse into a low-quality mode.
• Architecture-Dependent Optimization: The impact of design choices varies by model. For example, an invalidity penalty is essential for maintaining output diversity in the Qwen model but is redundant for Llama. Similarly, SFT initialization is indispensable for Llama but can be counterproductive for Qwen, where it may limit the model's ability to explore semantically diverse outputs.

This research demonstrates that reward function design is a primary determinant of success in structured generation tasks. It provides a practical framework for practitioners to improve LLM outputs without human preference data, while cautioning that universal defaults do not exist; reward strategies must be empirically tuned to the specific model architecture.