HDF5 notes

killer features

• hierarchical groups
• attributes
  
  • descriptive metadata
• slicing
  
  • actural data is on disk, slicing made it red to memroy
• have control of storage allocated
• support compression

HDF5

• large numerical arrays of homogenous type
• organized hierarchically
• tagging with arbitrary metadata
• high performance
• partial I/O

HDF5 data model

• dataset: array like objects that sotre numerical data on disk
  
  • attributes: name, type, shape
  • support random access
• group: hierarchical containers that store datasets and other groups
  
  • using B-trees
• attribute: user defined metadata, can be attached to dataset and group

HDF5 library

• written in C
• with C++, Java and Python bindings

read operation

1. h5py figures out the shape (10, 50) of the resulting array object.
2. An empty NumPy array is allocated of shape (10, 50).
3. HDF5 selects the appropriate part of the dataset.
4. HDF5 copies data from the dataset into the empty NumPy array.
5. The newly filled in NumPy array is returned.

write operation

1. h5py figures out the size of the selection, and determines whether it is compatible with the size of the array being assigned.
2. HDF5 makes an appropriately sized selection on the dataset.
3. HDF5 reads from the input array and writes to the file.

performance tips

• reduce read/write on the dataset

reshape

• can’t change the number of axes

Reference

1. How to access HDF5 data from Python
2. HDF5 for Python