Dataset Structure

The dataset includes 14 recording sessions, each stored separately in a Session_ID folder. Each folder contains its recorded cooking activities (e.g., cereal_bowl and tea). Each cooking activity folder contains two folders: dataset/ and Annotations/, which store the data and annotations, respectively.

dataset/ stores the synchronized data streams:

• Stereo events from the stereo Prophesee Gen4 cameras under LeftEvent/ and RightEvent/.
• Stereo RGB videos and global timestamps per RGB frame from the stereo CMOS cameras under LeftRGB/ and RightRGB/.
• RGB video and global timestamps per RGB frame from the RealSense D435i/D455 camera under DRGB/.
• Depth video and global timestamps per depth frame from the RealSense D435i/D455 camera under DEPTH/.
• 6-axis IMU data from the RealSense D435i/D455 camera under DIMU/.
• Note: We also provide additional 9-axis IMU data in IMU/, but it is not calibrated.

Annotations/ stores labels for object detection and action recognition:

• Global action segments are saved in actions/cooking_activity_action.csv.
• Bounding boxes on the left event camera are saved in objects/cooking_activity_LeftEvent_bbox.csv.
• Bounding boxes on the right event camera are saved in objects/cooking_activity_RightEvent_bbox.csv.
• Note: We also provide bounding boxes on the RealSense D435i/D455 camera in objects/cooking_activity_DRGB_bbox.csv.
• Please refer to the EventKitchen toolkit for more annotation details.

The detailed dataset structure is shown below:

Session_ID/
├── cooking_activity_01 (e.g. cereal_bowl)/
│   ├── dataset/
│   │   ├── DEPTH/
│   │   │   ├── DEPTH.avi
│   │   │   └── DEPTH_timestamp.csv
│   │   ├── DIMU/
│   │   │   └── DIMU.csv
│   │   ├── DRGB/
│   │   │   ├── DRGB.mp4
│   │   │   └── DRGB_timestamp.csv
│   │   ├── IMU/
│   │   │   └── IMU.csv
│   │   ├── LeftEvent/
│   │   │   └── LeftEvent.hdf5
│   │   ├── LeftRGB/
│   │   │   ├── LeftRGB.mp4
│   │   │   └── LeftRGB_timestamp.csv
│   │   ├── RightEvent/
│   │   │   └── RightEvent.hdf5
│   │   └── RightRGB/
│   │       ├── RightRGB.mp4
│   │       └── RightRGB_timestamp.csv
│   └── Annotations/
│       ├── actions/
│       │   └── cooking_activity_action.csv
│       └── objects/
│           ├── cooking_activity_DRGB_bbox.csv
│           ├── cooking_activity_LeftEvent_bbox.csv
│           └── cooking_activity_RightEvent_bbox.csv
├── cooking_activity_02/
│   └── ...
└── ...