Feature Attribution-Based Explainability Analysis of Deep Learning Models in Predictive Process Monitoring

Predictive process monitoring (PPM) uses deep learning to forecast business process outcomes, but these models are often black boxes. Existing explainability methods struggle to balance the trade-off between fragmented event-level explanations and overly broad trace-level summaries. This paper asks: how can we generate explainable, segment-level insights that respect the underlying control-flow dynamics of business processes?

The authors introduce a control-flow-aware segmentation algorithm that partitions event logs into meaningful subsequences. By using a Directly-Follows Relation (DFR) matrix, the method identifies 'change points' where the statistical properties of process transitions shift. These segments are then used as units for computing SHAP (Shapley Additive Explanations) values. This approach allows stakeholders to see which specific phases of a process—rather than individual events or entire traces—contribute most significantly to a model's prediction.

Findings

The proposed method successfully identifies process phase boundaries that align with known process logic in synthetic datasets. By grouping activities into coherent segments, the authors demonstrate that they can provide more intuitive explanations for real-world loan application and administrative processes. This segmentation not only improves the interpretability of the model's decision-making process but also reduces the computational complexity of calculating SHAP values compared to event-level approaches.

In high-stakes business environments, knowing that a model predicts a specific outcome is insufficient; practitioners need to understand the 'why' behind the prediction. By aligning explainability methods with the actual structure of business processes, this research bridges the gap between complex deep learning models and the practical requirements of process mining, fostering greater trust and adoption of AI in operational settings.