A Modular Workflow for Multimodal Reading Experiments

This paper presents a flexible, web-based experimental framework designed to capture and analyze multimodal data during naturalistic online reading. By integrating eye tracking, EEG, and interaction logs (mouse/keyboard), the workflow allows researchers to study cognitive processes—such as selective exposure or comprehension—in ecologically valid settings. A key innovation is the system's ability to process these streams in real time, enabling the experiment to dynamically trigger follow-up tasks (like ratings or labeling) based on a participant's physiological or behavioral responses during the session.

The workflow is built around the Lab Streaming Layer (LSL), which acts as the backbone for temporal synchronization across disparate sensors. The pipeline follows a three-phase structure:

1. Pre-experiment: Calibration and baseline measurements.
2. LSL Recording: Synchronized data collection during naturalistic reading tasks.
3. Post-experiment: Targeted feedback where the system uses metrics derived from the recording phase (e.g., fixation duration or EEG theta power) to prompt participants for qualitative input on specific sentences or areas of interest (AOIs).

By mapping gaze data directly to the browser's Document Object Model (DOM), the system links physiological signals to specific linguistic content, allowing for fine-grained analysis of how neural and behavioral responses correlate with textual characteristics.

Traditional reading research often relies on self-report measures, which are prone to bias and may not accurately reflect cognitive states during real-world digital consumption. This workflow bridges the gap between controlled laboratory experiments and naturalistic behavior. By providing a reusable, extensible architecture, it enables researchers to validate cognitive constructs—like selective exposure or emotional engagement—using objective, multimodal physiological data rather than relying solely on subjective post-hoc surveys.