Download Dataset

The EventKitchen dataset is hosted by 4TU.ResearchData with Delft University of Technology. You can access the dataset through 4TU.ResearchData below. There are 14 recording sessions in the dataset. Each session is compressed as a zip file named Kitchen_ID.zip. Download all zip files and unzip them.

4TU.ResearchData

We provide the EventKitchen toolkit to read the data and annotations.

EventKitchen Toolkit

Attention! The compressed dataset is around 900 GB. Please make sure you have at least 2 TB of free storage space to download and unzip it.

Dataset Structure

The dataset includes 14 recording sessions that are saved as Session_ID separately. Each Session_ID folder contains its recorded cooking activities (e.g. cereal_bowl, tea). Each cooking activity folder contains two folders: dataset/ and Annotations/, which store the data and annotations respectively.

dataset/ stores the synchronized data streams:

  • Stereo events from the stereo Prophesee Gen4 cameras under LeftEvent/ and RightEvent/.
  • Stereo RGB videos and global timestamps per RGB frame from the stereo CMOS cameras under LeftRGB/ and RightRGB/.
  • RGB video and global timestamps per RGB frame from the RealSense D435i/D455 camera under DRGB/.
  • Depth video and global timestamps per depth frame from the RealSense D435i/D455 camera under DEPTH/.
  • 6-axis IMU data from the RealSense D435i/D455 camera under DIMU/.
  • Note: we also provide additional 9-axis IMU data in IMU/, but it is not calibrated.

Annotations/ stores labels for object detection and action recognition:

  • Global action segments are saved in actions/cooking_activity_action.csv.
  • Bounding boxes on the left event camera are saved in objects/cooking_activity_LeftEvent_bbox.csv.
  • Bounding boxes on the right event camera are saved in objects/cooking_activity_RightEvent_bbox.csv.
  • Note: we also provide bounding boxes on the RealSense D435i/D455 camera in objects/cooking_activity_DRGB_bbox.csv.
  • Please refer to the EventKitchen toolkit for more annotation details.

The detailed dataset structure is shown below:

Kitchen_ID/
├── cooking_activity_01 (e.g. cereal_bowl)/
│   ├── dataset/
│   │   ├── DEPTH/
│   │   │   ├── DEPTH.avi
│   │   │   └── DEPTH_timestamp.csv
│   │   ├── DIMU/
│   │   │   └── DIMU.csv
│   │   ├── DRGB/
│   │   │   ├── DRGB.mp4
│   │   │   └── DRGB_timestamp.csv
│   │   ├── IMU/
│   │   │   └── IMU.csv
│   │   ├── LeftEvent/
│   │   │   └── LeftEvent.hdf5
│   │   ├── LeftRGB/
│   │   │   ├── LeftRGB.mp4
│   │   │   └── LeftRGB_timestamp.csv
│   │   ├── RightEvent/
│   │   │   └── RightEvent.hdf5
│   │   └── RightRGB/
│   │       ├── RightRGB.mp4
│   │       └── RightRGB_timestamp.csv
│   └── Annotations/
│       ├── actions/
│       │   └── cooking_activity_action.csv
│       └── objects/
│           ├── cooking_activity_DRGB_bbox.csv
│           ├── cooking_activity_LeftEvent_bbox.csv
│           └── cooking_activity_RightEvent_bbox.csv
├── cooking_activity_02/
│   └── ...
└── ...

Download Calibration

We provide intrinsic and extrinsic calibrations among sensors, which is saved in a zip file namely Calibration.zip in the 4TU.ResearchData. As we used IntelRealSense D435i and D455 for the data collection seperately, there are two sets of calibration results in the zip file. The correspondence of the calibration set and recording session is listed below:

  • D435i: Session 1, Session 2, Session 3, Session 4, Session 5 (coffee, tea, vege_salad, cut_cake), Session 6.
  • D455: Session 5 (fry_bacon, fry_pepper, fry_egg), Session 7, Session 8, Session 9, Session 10, Session 11, Session 12, Session 13, Session 14.
  • Attention! Be careful that the sequences in Session 5 are collected with two sets of sensors, as listed above.
The calibrations are all done with OpenCV. We provide the code in the EventKitchen toolkit to load calibration results. For more calibration details, please refer to the paper.