Parallelise HDF5 data loading
If a large number of files is to be loaded, I have the impression that speed of the DataManager is not limited by I/O but by CPU.
Proposal
Adjust the HDF5 data loader to load in multiple threads. Each thread would be responsible of loading a single file.
How to test the implementation?
Regular testing framework. Need to ensure thread-safety of the loaded objects; this boils down to whether they can be serialised or not; should not be an issue in our case.