Christopher Potts, Moritz Sudhof
8 min
Abstract
AI systems fail silently far more often than they fail visibly. In a large-scale quantitative analysis of human-AI interactions from the WildChat dataset, we find that 78% of AI failures are invisible: something went wrong but the user gave no overt indication that there was a problem. These invisible failures cluster into eight archetypes that help us characterize where and how AI systems are failing to meet users' needs. In addition, the archetypes show systematic co-occurrence patterns indicating higher-level failure types. To address the question of whether these archetypes will remain relevant as AI systems become more capable, we also assess failures for whether they are primarily interactional or capability-driven, finding that 91% involve interactional dynamics, and we estimate that 94% of such failures would persist even with a more capable model. Finally, we illustrate how the archetypes help us to identify systematic and variable AI limitations across different usage domains. Overall, we argue that our invisible failure taxonomy can be a key component in reliable failure monitoring for product developers, scientists, and policy makers. Our code and data are available at https://github.com/bigspinai/bigspin-invisible-failure-archetypes
Alex: Wait, so these acceptable chats seem fine on the surface, but they're loaded with signals? Give me an example of what that looks like.
Sam: Picture a developer asking for code, and the AI delivers something that runs but uses outdated commands no one should touch anymore. Or someone requests a quick business report with made-up references, and it hands over a longer version than asked—and the user just takes it. These slip through because the goal sort of gets met, but the AI's being sloppy in ways that add up.
Alex: Huh, so the acceptable tier is like a sneaky problem area—success masks the flaws. And how do these signals tie into overall quality?
Sam: Conversations rated good usually have zero or just one or two signals—90 percent have none at all. But as quality drops to poor or critical, signals pile up—almost all chats there have multiple negative ones, such as the AI drifting off track or acting overly sure when wrong. Signal count alone ends up being a solid clue to spot trouble.
Alex: So signal density flags the trouble spots pretty reliably. But with those eight archetypes for invisible failures, do they vary in how badly they tank quality?
Sam: Yes, they do. Most archetypes strongly link to poor or critical ratings—conversations that miss the mark badly or give misleading info. For instance, the silent mismatch never shows up in good chats and has the highest share of poor ones among invisible types; it's like the AI quietly delivers something off-base that sort of fits but doesn't really help.
Alex: Huh, so some patterns are sneakier failures. What about the common ones, like drift or confidence trap—do they predict low quality too?
Sam: They do, and they're especially telling because they're so frequent. Both heavily predict poor or critical outcomes, meaning when the AI veers to a related but wrong goal or states fiction with total assurance, the chat usually ends up subpar.
Alex: Interesting—users just bail when it's drifting nowhere. Do these archetypes ever team up, or do they happen solo?
Sam: They often overlap in telling ways. The researchers measured how much certain pairs stick together more than chance. They call this measure PPMI, for positive pointwise mutual information; darker spots on their chart highlight strong links, such as confidence trap with self-contradiction, or drift leading to walkaways.
Alex: Oh, so confident errors unravel into contradictions, or drifts make users quit. That points to bigger failure chains. Because models keep getting sharper, how did they test if these stick around?
Sam: They reran tags on a fresh sample using newer AIs like Claude Opus—distributions matched closely. Earlier, Claude Opus tagged 91% of failures as interaction-driven, not just smarts gaps, estimating 94% would linger in upgraded models. The paper suggests these dynamics are structural, evading fixes from capability alone.
Alex: So these patterns stick around because they're about conversation habits, not just raw brainpower. How exactly did they separate those—interaction issues from straight-up knowledge gaps?
Sam: They took over a thousand transcripts where the AI clearly missed the goal, then had Claude Opus check each one step by step. First, it confirmed if there really was a problem—95 percent passed that test. Then it asked: did the original AI lack the facts, logic, or skills to do the job? Or did the flop come from chat behavior, such as spitting out text instead of saying "I need more details"?
Alex: Okay, so capability is "can't do it," interaction is "won't ask or check right." What did Claude Opus decide overall?
Sam: In 91 percent of cases, interaction played a big role—58 percent mostly that, 33 percent a mix. Only 7 percent were mainly capability. And 94 percent of these would likely hang on even in smarter AIs.
Alex: Huh, so chatting style causes most trouble, and it won't vanish easily. Pulling it all together, though—what's the big picture here on why these patterns matter so much?
Sam: The core insight is that 91 percent of failures trace back to how AI handles conversations—persistent habits like over-generating or drifting—rather than just gaps in knowledge or smarts. These hold steady even as models improve, pointing to a need for fixes in interaction design over raw power alone. This shifts monitoring from end results to real-time tracking of archetype signals across chats.
Alex: Right, so conversation flow is the real weak spot. Does that mean developers could catch millions of these daily issues before users notice?
Sam: Exactly—this taxonomy applies to any set of human-AI chats, letting teams flag over 20 million invisible failures a day in active systems. It guides targeted changes, like UI tweaks to prompt clarifications or behavioral rules to curb drift.
Alex: Huh, that scales up monitoring without needing every user to report problems. But to keep it balanced, what are the limits to this approach?
Sam: A few key ones stand out. The signals were tuned for English text chats from WildChat, so voice, images, or other languages would need adjustments. Quality ratings rely on classifier judgments and team reviews, not direct user feedback. The data captures one snapshot of AI abilities, so exact rates may shift with newer models—though the patterns seem more lasting.
Alex: Fair points—they keep the findings grounded. So overall, this maps out a hidden layer of AI chats worth watching closely.
Sam: It does. By spotlighting these durable interaction pitfalls, the work offers a clear path to more reliable human-AI exchanges. That's the meaningful contribution here.
Alex: Thanks, Sam—solid breakdown. And that's our look at invisible failures in AI interactions. Thanks for listening to ResearchPod.