Perceived System Predictability: Scale Development and Application

How do users perceive the predictability of interactive systems, and how does this perception influence their trust and reliance? While objective predictability—the demonstrated ability to anticipate system outcomes—is frequently measured, HCI lacks a validated instrument for the subjective experience of predictability. The authors introduce Perceived System Predictability (PSP) as a construct grounded in uncertainty theory, distinguishing between epistemic (lack of knowledge), aleatory (inherent randomness), and effective predictability. They develop a 6-item scale through expert review and cognitive interviews, validating it across two studies ($N=200$ each) involving shape and sentiment classification systems.

The authors demonstrate that PSP is a robust, unidimensional construct that can also be interpreted through its three hierarchical facets. Key findings include:

• Dissociation of Measures: PSP and objective prediction correctness capture distinct aspects of a user's mental model. Objective performance metrics (like task completion time or correctness) do not predict PSP, suggesting that subjective human evaluation is essential for understanding user-system interaction.
• Predictive Power: Higher PSP scores correlate with increased objective prediction correctness, whereas higher trust scores are paradoxically associated with lower correctness, suggesting that trust and predictability are not interchangeable.
• Impact of Explanations and Stochasticity: Explanation formats (e.g., saliency maps vs. bar charts) significantly shift PSP ratings but do not improve objective correctness. Conversely, increasing system stochasticity degrades objective correctness without lowering PSP, indicating that users may suffer from an illusion of explanatory depth.

This research provides a principled foundation for designing transparent and trustworthy AI. By measuring PSP, researchers can identify when users are over-relying on systems or when their mental models are misaligned with actual system behavior. The validated scale offers a practical tool for practitioners to assess whether their systems are truly predictable, rather than just appearing so.