Max Schemmer, Niklas Kühl, Carina Benz, Andrea Bartos, Gerhard Satzger
5 min
As AI systems become more prevalent in high-stakes decision-making, researchers have shifted focus from simply maximizing AI adoption to achieving appropriate reliance—the ability of a human to follow correct AI advice and reject incorrect AI advice. This paper addresses the lack of a standardized definition and measurement for this behavior and investigates whether providing explanations for AI advice helps users achieve this goal.
The authors introduce a two-dimensional, quantifiable metric called Appropriateness of Reliance (AoR). This metric consists of two components: Relative AI Reliance (RAIR), which measures how often users correctly switch to AI advice when their initial decision was wrong, and Relative Self-Reliance (RSR), which measures how often users correctly ignore AI advice when it is incorrect. The authors conducted a behavioral experiment with 200 participants using a hotel review deception detection task. Participants made an initial decision, received AI advice (with or without feature-importance explanations), and then had the opportunity to revise their decision.
The study reveals that providing feature-importance explanations significantly increases RAIR, suggesting that explanations help users learn from the AI when they lack domain knowledge. However, these explanations had no significant effect on RSR, meaning they did not help users better identify when the AI was wrong. The researchers also found that while trust in the AI correlates with both RAIR and RSR, the effect of explanations on RAIR is partially mediated by changes in the user's self-confidence after seeing the AI's input.
This work provides a rigorous, standardized framework for evaluating human-AI interaction. By distinguishing between the ability to accept good advice and reject bad advice, the AoR metric allows designers to pinpoint exactly where human-AI teams are failing. The findings suggest that current explainable AI (XAI) techniques may be more effective at teaching users than at helping them critically evaluate AI errors, highlighting a need for new design strategies to mitigate over-reliance.
AI advice is becoming increasingly popular, e.g., in investment and medical treatment decisions. As this advice is typically imperfect, decision-makers have to exert discretion as to whether actually follow that advice: they have to "appropriately" rely on correct and turn down incorrect advice. However, current research on appropriate reliance still lacks a common definition as well as an operational measurement concept. Additionally, no in-depth behavioral experiments have been conducted that help understand the factors influencing this behavior. In this paper, we propose Appropriateness of Reliance (AoR) as an underlying, quantifiable two-dimensional measurement concept. We develop a research model that analyzes the effect of providing explanations for AI advice. In an experiment with 200 participants, we demonstrate how these explanations influence the AoR, and, thus, the effectiveness of AI advice. Our work contributes fundamental concepts for the analysis of reliance behavior and the purposeful design of AI advisors.
Sam: Which is the crux of it. What do they find? [[RP_SECTION:dual-effect-of-transparency|Dual Effect of Transparency]]
Alex: A dual effect, and the asymmetry is what's worth paying attention to. Explanations do support Relative AI Reliance—users are better at recognizing when they should update toward a correct AI recommendation. But on the Relative Self-Reliance axis, the picture is worse. Explanations also increase over-reliance: users follow incorrect AI advice more often when an explanation is present, because the explanation makes the wrong answer seem plausible. The mechanism is that feature importance outputs provide a locally coherent narrative, and users read that narrative as evidence of correctness rather than as a description of the model's reasoning process.
Sam: So the explanation is doing exactly what it's designed to do—communicating the model's internal logic—but users are treating that communication as a quality signal rather than an informational one. The transparency itself becomes the confound.
Alex: Right. And this is where the sequential task design becomes a genuine limitation worth naming. By forcing an initial judgment before the AI recommendation appears, the paradigm may actually be suppressing the over-reliance it's trying to measure. In a real deployment, there's often no enforced prior commitment—users might anchor on the AI's output from the start, which would push Relative Self-Reliance scores down considerably. The RSR results here might represent something closer to an optimistic ceiling rather than an ecologically valid estimate.
Sam: So the effect sizes on the self-reliance axis could be understated in precisely the direction that matters most for high-stakes domains. That's a meaningful constraint on generalizability.
Alex: It is. The authors are candid about this—they flag non-sequential paradigms as a necessary direction for follow-on work. That said, the framework itself doesn't depend on the sequential design. AoR as a metric is portable to other paradigms, and that portability is probably the most durable contribution here. We're moving from asking whether users accept the AI, to asking whether they're actually better decision-makers with it. In any domain where errors carry real costs—clinical diagnosis, financial risk assessment, infrastructure monitoring—that's the only question that matters.
Sam: And if you can track those two axes in real time, you could imagine the system adapting—modulating how it presents explanations, or withholding them, specifically when a user appears most susceptible to over-reliance. [[RP_SECTION:adaptive-explanation-systems|Adaptive Explanation Systems]]
Alex: That's exactly where the authors point. It's a meaningful shift from static explanation design toward explanation-as-intervention. Rather than deciding once at design time whether to show a SHAP plot, the system monitors calibration continuously and adjusts accordingly. The framework gives you the measurement infrastructure to make that kind of adaptive loop tractable.
Sam: It's a more honest accounting of what human-AI complementarity actually requires. Not just transparency, but the scaffolding to help users use that transparency well.
Alex: Precisely. And the fact that the same explanations that improve appropriate reliance also worsen over-reliance is a result that should give pause to anyone deploying explainable AI in consequential settings. The intervention and the failure mode are the same artifact. Thanks for listening to ResearchPod.