Modeling Adaptive Visual Search in Semantically Hierarchical Layouts

How do humans adapt their visual search strategies to the hierarchical structure of information in user interfaces? Specifically, the authors investigate how the alignment of semantic categories with spatial groupings allows users to optimize their search behavior despite the inherent capacity limitations of visual short-term memory (VSTM).

The researchers developed a computational cognitive model based on the framework of computational rationality. Unlike traditional models that rely on fixed, predefined rules for eye movements, this model treats the user as an agent learning an optimal search policy. The agent operates within a partially observable Markov decision process (POMDP), where it must maximize search efficiency by deciding where to fixate next. The model incorporates a hierarchical VSTM, allowing the agent to "chunk" visual elements into groups. When a group is identified as semantically irrelevant to the target, the agent can inhibit the entire group, effectively bypassing it to save time.

The model successfully replicates human task durations and eye movement patterns observed in two experimental studies. The findings confirm that search efficiency is significantly improved when semantic grouping aligns with spatial layout. When semantic consistency is high, the model learns to "jump" over entire visual regions that do not match the target category. Conversely, when semantic information is absent or inconsistent, the model reverts to an exhaustive, methodical search strategy, demonstrating that human search behavior is a flexible adaptation to the specific constraints of the environment.

This research provides a powerful tool for UI designers to evaluate how different layout strategies impact user performance. By moving beyond empirical, descriptive models toward an explanatory, cognitively grounded framework, this work allows designers to simulate how users will navigate complex interfaces before they are even built. It highlights the importance of aligning visual design with the semantic mental models of users to facilitate faster and more intuitive information retrieval.