Gaze-Informed Proactive AI Assistance for Children’s Picture Exploration

Proactive AI assistance—systems that offer support without explicit user prompts—holds great potential for children, whose needs are often expressed through nonverbal behaviors like eye gaze rather than verbal queries. This study investigates how to design such assistance for children’s picture exploration, a key activity for visual literacy and narrative development. The authors developed Ollie, a gaze-informed AI assistant that uses eye-tracking data to determine when to intervene and what to describe. Ollie estimates a child's attentional state using a Hidden Markov Model (HMM) and selects contextually relevant picture regions to narrate, aiming to guide the child's exploration naturally.

The researchers conducted a within-subject experiment with 22 children, comparing Ollie’s gaze-informed assistance against a random-assistance baseline. The results demonstrate that gaze-informed assistance is superior in several key metrics: it sustained children's attention on the primary focus area for significantly longer and guided them more effectively toward related picture regions. While the assistance did not significantly alter the overall quality or content of children's verbal descriptions after exploration, both children and their parents/teachers preferred the gaze-informed approach, noting that it felt more aligned with the child's interests and provided a more coherent exploration experience.

This work provides a blueprint for using gaze as an implicit input for proactive AI, moving beyond reactive, command-based interactions. By grounding AI narration in a child's moment-to-moment visual attention, systems can provide timely, relevant support that respects the child's natural exploration process. This approach is particularly promising for open-ended, visually rich activities where children may struggle to articulate their needs, offering a path toward more intuitive, child-centered AI companions.