Skip to content

Improve DAG and plotting performance and experience

Utopia Developers requested to merge improve-dag into master

This MR adds several improvements to DAG-based plotting.

Performance improvements

The following measures lead to a ~40% speedup of building the TransformationDAG. This is only relevant for many nodes (several 100k), in which case constructing the DAG sometimes takes some time (roughly 30s for 500k nodes on my machine).

  • Use cPickle-based deepcopy in DAG, which is much faster. If that fails, can still fallback to copy.deepcopy
  • Reduce unnecessary logging

Communication improvements

  • Let PlotManager communicate the number of plots and the time needed to execute them
  • Let TransformationDAG give useful error messages upon missing tags or references

New features

  • #281 (closed), UniversePlotCreator now allows plotting specific universes by their ID.

Other improvements and changes

  • Plots from a plot configuration are continued even if one creator failed during config preparation
  • Extend print_data operation to allow using format strings
  • Hash function now also works for byte objects (was unnecessarily restricted before)
  • Let XrDataContainer rename the underlying data only if it has no name; and adjust DAG cache loading to unpack xarray objects, which should not be wrapped in dantro tree objects – this addresses #275 (closed)
  • Better error messages in PlotManager
  • Better DataOperationFailed error (#276 (closed))
  • Exclude "private" tags from compute_only: all (#272 (closed))
  • Add raise_SkipPlot data operation for plot skipping; addresses #165 (closed)

Can this MR be accepted?

  • Implementation ready
  • Tests added or adjusted
  • Documentation extended or updated
  • Code quality
  • Ready for merging
    • Pipeline passes without warnings
    • History cleaned-up or squash option set
    • Changelog entry added
    • Version number bumped
    • Reviewed & approved by @

Related issues

#251 (closed)

Closes #281 (closed), #275 (closed), #276 (closed), #272 (closed), #165 (closed)

Edited by Utopia Developers

Merge request reports