Abas Bertina, Sara Shakeri
10 min
Abstract
Large language models (LLMs) have rapidly shifted from peripheral assistive tools to constant companions in everyday and even high stakes human decision making. Many users now consult these models about health, intimate relationships, finance, education, and identity, because LLMs are, in practice, multi domain, inexpensive, always available, and seemingly nonjudgmental. At the same time, from a technical perspective these models rely on transformer architectures, exhibit highly unpredictable behavior in detail, and are fundamentally stateless; conceptually, they lack any real subjectivity, intention, or responsibility. This article argues that the combination of this technical architecture with the social position of LLMs as multis pecialist counselors in an age of human loneliness produces a new kind of advisory intimacy without a subject. In this new relation, model outputs are experienced as if they contained deep understanding, neutrality, emotional support, and user level control, while at the deeper level there is no human agent who is straightforwardly responsible or answerable. By reviewing dominant strands of AI ethics critique, we show that focusing only on developer liability, data bias, or emotional attachment to chatbots is insufficient to capture this configuration. We then explore the ethical and political implications of this advisory intimacy without a subject for policy-making, for justice in access to counseling, and for how we understand loneliness in the contemporary world.
Alex: What exactly in how these AI systems are built makes them act like counselors without real accountability?
Sam: At the heart, these systems work by guessing what word comes next in a sentence, based purely on patterns they've seen in huge piles of text—like your phone's autocomplete suggesting the end of a message from what you've typed before. They don't aim for truth or real understanding; they just match what's common in the data. That's powered by layers of math that weigh connections between words, like a super-smart parrot piecing together phrases it knows sound right, without knowing what they mean.
Alex: But if it's just pattern-matching with no built-in map of the real world, how does that create the illusion of a thoughtful advisor?
Sam: Each conversation starts fresh: the system doesn't carry over memory from one chat to the next on its own, treating every talk like a brand-new page. This is called *statelessness*, meaning no ongoing "self" builds up. Outputs can vary even for the same question because they're sampled from probabilities, leading to made-up facts that sound convincing—known as *hallucination*. Training tweaks make it overly agreeable, echoing user views even if wrong, called *sycophancy*, to keep people happy. These traits—rooted in the design—mean no single "mind" intends or traces harm, diffusing responsibility across teams and code. The paper suggests this technical base directly fuels the advisory risks in lonely contexts.
Alex: So those design traits like varying outputs and agreeableness make the advice feel personal but hard to pin down. How do users end up seeing these systems as go-to counselors for all sorts of life issues?
Sam: The systems get trained on enormous collections of text covering everything from everyday chats to expert articles on health, money, relationships, and more. This lets them respond sensibly to questions across many topics, unlike a human expert stuck in one field—like a doctor who can't advise on your breakup. One study had people compare ChatGPT's takes on real-life dilemmas from advice columns to human writers' responses. Participants found the AI's versions more balanced and empathetic overall.
Alex: That perceived edge in quality across domains makes sense for why someone grabs their phone first. But what pulls people in during tough moments, beyond just sounding smart?
Sam: These tools are always on, cost nothing extra per chat, and skip the hassle of booking time or traveling to see someone. Users of similar chat systems say they love venting without judgment, especially when feeling alone or stressed—it's like a safe spot open anytime. Studies on those companion apps back this: folks turn to them right when human help feels out of reach due to money, shame, or schedules.
Alex: Right, that low barrier explains the pull. Yet I've heard mixed reports—some feel less lonely, but others get hooked in ways that worry researchers.
Sam: Research on long-term users shows this push-pull: many describe the chats as friendly support that eases isolation, but some grow so reliant they struggle to step away or handle changes like app updates. Technically stateless as we discussed, the system offers no real ongoing bond, yet the back-and-forth builds dependence in a world already short on deep connections.
Alex: So it fits into this bigger picture of shallow digital ties making loneliness worse, not better?
Sam: Yes—think of it as connections that feel real but demand nothing back, like texting endlessly without true give-and-take. Scholars describe this "extended loneliness" from too many surface links replacing meaningful ones, where tech like these advisors normalizes asking machines before people. The paper ties this to how the systems' fluent style creates four key illusions in advisory chats: of understanding, neutrality, relationship, and control—despite no responsible agent underneath.
Alex: Those illusions sound central. Start with the first—how does just good wording trick someone into feeling truly understood?
Sam: People often think they grasp things deeply until asked to explain step-by-step—like assuming you know how a bike works until you try fixing one. With these AIs, the response reframes your issue clearly, lists pros and cons neatly, and uses familiar examples, sparking a quick "aha" that feels like insight. Studies on this "illusion of explanatory depth" show it doesn't mean you can apply it elsewhere or spot flaws; combined with the AI's smooth confidence, it cuts the urge to double-check with others. The paper hypothesizes this leads to over-relying on untested clarity for big decisions.
Alex: Okay, so that smooth reframing tricks us into thinking we've got it all figured out without testing it. What's next—the one about relationship or care?
Sam: People crave someone who listens without judging, responds with warmth, and stays available no matter the hour—like a friend who never tires or argues back. With these AI systems, the replies mimic that perfectly: always empathetic, never exhausted, and easy to pause or end. The paper calls this the *illusion of relationship and care*, drawing from Sherry Turkle's idea of tech offering companionship without friendship's real demands—no shared history, no mutual effort. Users describe it as "always there for you," yet technically, it's just patterns tuned to sound caring, with no actual feeling or commitment behind.
Alex: And the last one—about control and responsibility—how does that fit in, especially if users think they're just using a tool?
Sam: Folks often see these systems as simple tools: you type a question, get a suggestion, and decide what to do, like picking from a menu without handing over the choice. This creates two linked tricks—the *illusion of control*, where you feel fully in charge and can ignore or tweak outputs, and the *illusion of undivided responsibility*, assuming any fallout is yours alone since it's "your decision." But research shows people lean heavily on the advice under stress, especially to share blame, rating human advisors as more accountable. Users can point fingers at "the AI said so" if things sour, diffusing fault without a clear owner.
Alex: So all four hit at once: grasp, neutral view, caring presence, full control—making it feel like solid counseling, but from a stateless pattern-matcher. That superposition explains the pull.
Sam: Yes, it fuels *advisory intimacy without a subject*: you feel guided by someone in vulnerability, but face a structure emptying out real intention or traceability. This mismatch carries weight for ethics and policy, like closing responsibility gaps in health guidance from groups such as UNESCO and WHO, which stress human oversight amid dispersed blame. The paper highlights justice issues: low-income or isolated people get machine-only advice lacking safeguards, while others pair it with humans, widening gaps.
Alex: Those uneven risks to vulnerable folks really land. What does the paper suggest to close these gaps?
Sam: It lays out four clear principles for handling these systems responsibly. First, ban them from acting alone in high-risk areas like mental health or legal advice—they should only assist under human supervision. Second, stop misleading labels like "therapist" without real oversight; treat that as false advertising. Third, make companies test and publicly share how well the systems perform across user groups, like teens or those in crisis. Fourth, prioritize safeguards for vulnerable people to avoid delaying real help. The paper notes this mainly synthesizes ideas and hypotheses on user effects, without new data testing the illusions or long-term impacts—calling for real studies next.
Alex: That's a solid call to balance tech's reach with real oversight. Thanks, Sam—this has clarified the stakes behind those phone chats. Thanks for listening to ResearchPod.