Skip to content

Reduce memory usage

Utopia Developers requested to merge reduce-memory-usage into master

This MR aims to reduce dantro's memory usage through the following measures:

  • In dantro.utils.coords, use range objects instead of directly generating lists
    • This can and should be postponed as long as possible. Significantly reduces RAM use!
  • Remove _logstr cache attribute from AbstractDataContainer (and thereby all tree objects)
    • Recomputing the value when needed (not that often) takes ~100 ns, so it's really not worth caching
  • Share h5.File objects between instances (not done, not feasible)
    • Attempted to do this in 4f44778b and 1378f41b, but ultimately had to revert these because updating the shared object is prohibitively difficult when using the parallel loader

Can this MR be accepted?

  • Implementation ready
  • Tests added or adjusted
  • Documentation extended or updated
  • Code quality
  • Ready for merging

Related issues

#251 (closed)

Edited by Utopia Developers

Merge request reports