Conversational Tactile Data Interfaces: Co-Designing Accessible Data Experiences with Blind Users Using Refreshable Tactile Displays and Conversational AI

This paper presents Graphy, a conversational tactile data interface (CTDI) designed to bridge the gap between static tactile graphics and interactive data exploration for blind and low-vision (BLV) users. The researchers employed a longitudinal co-design process involving three BLV co-designers over eight months and four workshops. By moving beyond previous Wizard-of-Oz simulations, the team instantiated a functional system that integrates a multi-line refreshable tactile display (RTD) with an LLM-powered conversational agent. The goal was to determine how to effectively coordinate touch-based spatial exploration with verbal analytical support.

Graphy introduces several novel interaction patterns to support sensemaking:

• Layered Presentation: Instead of overwhelming the user with a full chart, the system scaffolds exploration by introducing chart components (e.g., axes, individual data series) one at a time, allowing users to build a mental model incrementally.
• Sequential Interaction Pattern: The system enforces a 'select, confirm, ask, verify' workflow. Users select data points via touch, confirm their selection through tactile and audio feedback, query the agent for analysis, and then use the tactile display to verify the agent's response against the raw data.
• Feedback Grammar: The system distinguishes between user-initiated actions and agent-initiated highlights, using synchronized tactile animations to draw the user's attention to specific data points referenced in the agent's spoken responses.

Accessible data visualization has historically struggled to balance spatial understanding with analytical depth. Static tactile graphics are difficult to update and lack interactivity, while audio-only descriptions often fail to convey the 'shape' of data. Graphy demonstrates that by treating touch as the primary channel for spatial sensemaking and the conversational agent as a tool for calculation and context, developers can create more independent and fluid data exploration experiences for BLV users. The design recommendations provided in the paper offer a roadmap for future development of multimodal interfaces that combine physical tactile feedback with generative AI.