Balancing Safety and Autonomy: Accessibility-Oriented Interventions in Generative AI for Cognitive Impairment

Generative AI systems are increasingly integrated into the daily routines of older adults with cognitive impairment, offering flexible language-based support for tasks like communication, information seeking, and health management. While these technologies help bridge accessibility gaps, their open-ended and probabilistic outputs introduce risks such as user over-reliance, misinterpretation, and inappropriate decision-making. Prior research has primarily examined usability and adoption, leaving a critical gap in understanding how system design shapes user agency, participation in decision-making, and power distribution within care networks.

To address this, the authors conducted a qualitative study involving 45 individuals with cognitive impairment and their caregivers. Using semi-structured interviews, the research investigates how accessibility-oriented interventions embedded within generative AI systems structure real-world interactions and influence the balance between user safety and personal autonomy across varying stages of cognitive decline.

The study identifies five recurring accessibility-oriented mechanisms through which generative AI systems structure interaction:

• AI Capability Constraint: Limiting system scope to prevent hazardous open-ended outputs.
• Human Oversight Embedding: Integrating caregiver or clinician supervision into the operational loop.
• Cognitive Engagement Maintenance: Prompting users to remain active participants in reasoning tasks rather than passive consumers.
• Human-AI Relationship Regulation: Shaping user perceptions of companionship, authority, and trust toward the system.
• Risk Transparency and Control: Exposing system uncertainty and offering mechanisms to mitigate errors.

These mechanisms operate through two main logics: enablement, which sustains user reasoning, and protection, which reduces risk by constraining system behavior and redistributing decision-making responsibilities.

The effects of these intervention mechanisms are not uniform; they vary significantly based on the user's level of cognitive impairment. For individuals with severe impairment, protective mechanisms provide essential safeguards that prevent critical mistakes. However, for users with mild impairment, the exact same protective interventions often act as restrictive constraints that diminish autonomy and limit meaningful participation.

Furthermore, as cognitive impairment progresses, tensions between safety and autonomy do not truly resolve. Instead, they become less visible because the user's capacity to perceive, question, or resist system constraints diminishes. This apparent improvement in system safety is frequently achieved at the cost of reduced user involvement, shifting cognitive burdens onto human caregivers, and altering expectations of independence.