ResearchPod Summary
As AI systems become more prevalent in high-stakes decision-making, researchers have shifted focus from simply maximizing AI adoption to achieving appropriate reliance—the ability of a human to follow correct AI advice and reject incorrect AI advice. This paper addresses the lack of a standardized definition and measurement for this behavior and investigates whether providing explanations for AI advice helps users achieve this goal.
The authors introduce a two-dimensional, quantifiable metric called Appropriateness of Reliance (AoR). This metric consists of two components: Relative AI Reliance (RAIR), which measures how often users correctly switch to AI advice when their initial decision was wrong, and Relative Self-Reliance (RSR), which measures how often users correctly ignore AI advice when it is incorrect. The authors conducted a behavioral experiment with 200 participants using a hotel review deception detection task. Participants made an initial decision, received AI advice (with or without feature-importance explanations), and then had the opportunity to revise their decision.
The study reveals that providing feature-importance explanations significantly increases RAIR, suggesting that explanations help users learn from the AI when they lack domain knowledge. However, these explanations had no significant effect on RSR, meaning they did not help users better identify when the AI was wrong. The researchers also found that while trust in the AI correlates with both RAIR and RSR, the effect of explanations on RAIR is partially mediated by changes in the user's self-confidence after seeing the AI's input.
This work provides a rigorous, standardized framework for evaluating human-AI interaction. By distinguishing between the ability to accept good advice and reject bad advice, the AoR metric allows designers to pinpoint exactly where human-AI teams are failing. The findings suggest that current explainable AI (XAI) techniques may be more effective at teaching users than at helping them critically evaluate AI errors, highlighting a need for new design strategies to mitigate over-reliance.
[[RP_SECTION:limitations-of-acceptance-metrics|Limitations of Acceptance Metrics]]
Alex: We've spent a decade optimizing AI systems for user acceptance, but we've largely neglected the complementary skill: knowing when to push back. A paper from Max Schemmer and colleagues at the Karlsruhe Institute of Technology, published at the International Conference on Intelligent User Interfaces, puts a rigorous measurement framework around that problem.
Sam: So the field has been chasing a metric that might actually be counterproductive? If you just maximize acceptance, you're training users to be passive receivers rather than critical collaborators.
Alex: That's the argument. Current evaluation focuses on binary acceptance—did the user follow the AI or not? But that ignores the quality of the decision. If the AI was wrong and the user still followed it, that counts as acceptance. If the AI was right and the user overrode it, that also counts as non-acceptance. Both get lumped together, which means the metric is blind to the thing we actually care about.
Sam: So it's a calibration problem, not a compliance problem. How do they operationalize that distinction? [[RP_SECTION:appropriateness-of-reliance-framework|Appropriateness of Reliance Framework]]
Alex: They propose a two-dimensional metric called Appropriateness of Reliance, or AoR. The first axis is Relative AI Reliance—how often you correctly update toward the AI when your initial answer was wrong. The second is Relative Self-Reliance—how often you correctly hold your ground when the AI is wrong. Those two axes are independent. You can score well on one and poorly on the other, which is exactly the point.
Sam: That decoupling is doing a lot of work. It means you can distinguish a user who's genuinely learning to recognize AI competence from one who's just deferring by default. What's the experimental setup? [[RP_SECTION:experimental-design-and-explanations|Experimental Design and Explanations]]
Alex: They use a human-AI decision task with a sequential structure: participants make an initial judgment, then see the AI's recommendation along with a local feature importance explanation, then make a final call. That sequencing is what lets them compute both axes—you need the pre-AI decision to know whether the post-AI decision was an appropriate update or not. They then use structural equation modeling to trace how the explanations influence each axis, with trust and self-confidence as mediators.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: And the explanations here are local feature importance—so something like LIME or SHAP outputs, highlighting which features drove the prediction?
Alex: Correct. The kind of explanation that tells you "the model weighted these three features most heavily for this case." The question is whether that transparency actually improves calibration, or whether it just makes the AI's output feel more authoritative regardless of whether it's right.
Sam: Which is the crux of it. What do they find? [[RP_SECTION:dual-effect-of-transparency|Dual Effect of Transparency]]
Alex: A dual effect, and the asymmetry is what's worth paying attention to. Explanations do support Relative AI Reliance—users are better at recognizing when they should update toward a correct AI recommendation. But on the Relative Self-Reliance axis, the picture is worse. Explanations also increase over-reliance: users follow incorrect AI advice more often when an explanation is present, because the explanation makes the wrong answer seem plausible. The mechanism is that feature importance outputs provide a locally coherent narrative, and users read that narrative as evidence of correctness rather than as a description of the model's reasoning process.
Sam: So the explanation is doing exactly what it's designed to do—communicating the model's internal logic—but users are treating that communication as a quality signal rather than an informational one. The transparency itself becomes the confound.
Alex: Right. And this is where the sequential task design becomes a genuine limitation worth naming. By forcing an initial judgment before the AI recommendation appears, the paradigm may actually be suppressing the over-reliance it's trying to measure. In a real deployment, there's often no enforced prior commitment—users might anchor on the AI's output from the start, which would push Relative Self-Reliance scores down considerably. The RSR results here might represent something closer to an optimistic ceiling rather than an ecologically valid estimate.
Sam: So the effect sizes on the self-reliance axis could be understated in precisely the direction that matters most for high-stakes domains. That's a meaningful constraint on generalizability.
Alex: It is. The authors are candid about this—they flag non-sequential paradigms as a necessary direction for follow-on work. That said, the framework itself doesn't depend on the sequential design. AoR as a metric is portable to other paradigms, and that portability is probably the most durable contribution here. We're moving from asking whether users accept the AI, to asking whether they're actually better decision-makers with it. In any domain where errors carry real costs—clinical diagnosis, financial risk assessment, infrastructure monitoring—that's the only question that matters.
Sam: And if you can track those two axes in real time, you could imagine the system adapting—modulating how it presents explanations, or withholding them, specifically when a user appears most susceptible to over-reliance. [[RP_SECTION:adaptive-explanation-systems|Adaptive Explanation Systems]]
Alex: That's exactly where the authors point. It's a meaningful shift from static explanation design toward explanation-as-intervention. Rather than deciding once at design time whether to show a SHAP plot, the system monitors calibration continuously and adjusts accordingly. The framework gives you the measurement infrastructure to make that kind of adaptive loop tractable.
Sam: It's a more honest accounting of what human-AI complementarity actually requires. Not just transparency, but the scaffolding to help users use that transparency well.
Alex: Precisely. And the fact that the same explanations that improve appropriate reliance also worsen over-reliance is a result that should give pause to anyone deploying explainable AI in consequential settings. The intervention and the failure mode are the same artifact. Thanks for listening to ResearchPod.