Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen

Neuromorphic vision, utilizing event cameras that respond asynchronously to intensity changes, has primarily focused on autonomous driving and drone navigation. While these sensors offer high temporal resolution and dynamic range, their application to human-centric daily activities remains underexplored. Existing datasets in this domain often rely on scripted actions in controlled laboratory settings or synthetic data, which fail to capture the complexity and spontaneity of real-world human behavior.

EventKitchen addresses these limitations by providing a large-scale, real-world, egocentric benchmark for neuromorphic perception. The dataset was collected using a wearable helmet system equipped with a stereo pair of Prophesee Gen4 event cameras, two CMOS RGB cameras, and an Intel RealSense depth camera. This multi-modal setup allows for the capture of 5.5 hours of unscripted cooking activities performed by 10 participants across 13 diverse kitchen environments.

To facilitate robust benchmarking, the authors provide extensive annotations, including 10,762 action segments and 13,482 bounding boxes for 12 common kitchenware categories. The dataset includes synchronized streams of stereo events, RGB video, depth maps, and IMU data. A key feature of the dataset is the use of a projection pipeline that maps human-annotated bounding boxes from the depth-aligned RGB frames onto the stereo event camera domains, ensuring high-quality ground truth for event-based tasks like object detection and stereo depth estimation.

EventKitchen moves neuromorphic vision research beyond the constraints of autonomous driving and scripted laboratory tasks. By providing a challenging, real-world dataset that reflects the variability of daily life, it enables the development and evaluation of more robust event-based perception models. The inclusion of stereo event data and ground-truth depth maps specifically supports advancements in 3D perception and human-object interaction analysis in wearable AI applications.