Maxwell : hdf5

Hierarchical Data Format 5 (HDF5) is a unique open source technology suite for managing data collections of all sizes and complexity.HDF5 was specifically designed:

• For high volume and/or complex data (but can be used for low volume/simple data) 
• For every size and type of system (portable) 
• For flexible, efficient storage and I/O 
• To enable applications to evolve in their use of HDF5 and to accommodate new models 
• To be used as a file format tool kit (many formats use HDF5 under the hood)

[Taken from the HDF5 introduction.]

Using  HDF5 libraries

The RedHat/EPEL provided HDF5 libraries install into standard system PATH (/usr/lib64) and will be found automatically by compiler/linker. To use the openmpi-compiled version of HDF5 you need to initialize the openmpi environment, for example by module load mpi/openmpi.In addition to the (comparably old) standard libraries we also provide a current implementation (V1.10.x) on EL7 systems. To use these libraries you need to load the correspong hdf5 modules:

o initialize the environment use the module command:

[elx]% module avail
[elx]% module load hdf5             # standard HDF5 
[elx]% module load hdf5/1.10.1      # to make sure that v1.10.1 is loaded
[elx]% module load hdf5-openmpi     # openmpi-compiled HDF5 

HDF5 plugins

HDF5 can be dynamically extended using a plugin mechanism. HDF5 uses the environmental variable HDF5_PLUGIN_PATH to locate plugins. Set/exapnd the variable to include your own custom plugins. Currently we only deploy the lz4 plugin located in /usr/lib64/hdf5/plugins. The environment is set a login in /etc/profile.d/hdf5-plugins.sh. This applies however not for CSH-users and /etc/profile.d is not parsed in non-interactive jobs. You might need to set the environment yourself in case it's missing:

[elx]% cat /etc/profile.d/hdf5-plugins.sh
export HDF5_PLUGIN_PATH=/usr/lib64/hdf5/plugins
[elx]% export HDF5_PLUGIN_PATH=/home/$USER/hdf5/plugins:/usr/lib64/hdf5/plugins    # expand the plugins PATH