Improve DAG and plotting performance and experience
This MR adds several improvements to DAG-based plotting.
Performance improvements
The following measures lead to a ~40% speedup of building the TransformationDAG
. This is only relevant for many nodes (several 100k), in which case constructing the DAG sometimes takes some time (roughly 30s for 500k nodes on my machine).
- Use cPickle-based deepcopy in DAG, which is much faster. If that fails, can still fallback to
copy.deepcopy
- Reduce unnecessary logging
Communication improvements
- Let
PlotManager
communicate the number of plots and the time needed to execute them - Let
TransformationDAG
give useful error messages upon missing tags or references
New features
-
#281 (closed),
UniversePlotCreator
now allows plotting specific universes by their ID.
Other improvements and changes
- Plots from a plot configuration are continued even if one creator failed during config preparation
- Extend
print_data
operation to allow using format strings - Hash function now also works for byte objects (was unnecessarily restricted before)
- Let
XrDataContainer
rename the underlying data only if it has no name; and adjust DAG cache loading to unpack xarray objects, which should not be wrapped in dantro tree objects – this addresses #275 (closed) - Better error messages in
PlotManager
- Better
DataOperationFailed
error (#276 (closed)) - Exclude "private" tags from
compute_only: all
(#272 (closed)) - Add
raise_SkipPlot
data operation for plot skipping; addresses #165 (closed)
Can this MR be accepted?
-
Implementation ready -
Tests added or adjusted -
Documentation extended or updated
-
Code quality -
Changes follow coding guidelines -
Checked code coverage on new and adjusted code
-
-
Ready for merging -
Pipeline passes without warnings -
History cleaned-up or squash option set -
Changelog entry added -
Version number bumped -
Reviewed & approved by @
-
Related issues
Closes #281 (closed), #275 (closed), #276 (closed), #272 (closed), #165 (closed)
Edited by Utopia Developers