Address concurrency and cache-reading issues with forced cache read/write
This MR aims to address issues raised by @peanutfun in #342 (closed).
-
Allow not to unpack loaded data cache objects (i.e. return cont.data
instead of thecont
the data was loaded into).- This is not a long-term solution, but a temporary workaround to customise whether loaded objects should again be dantro containers.
In the long run, this requires more intelligent saving of cache files that allow restoring into exactly the type that they were before.
read: enabled: true always: true load_options: unpack: false chunks: {...} # ... further options
- This is not a long-term solution, but a temporary workaround to customise whether loaded objects should again be dantro containers.
In the long run, this requires more intelligent saving of cache files that allow restoring into exactly the type that they were before.
-
Fixes a bug where the DataManager
acted as an unwanted cache layer despite being configured toalways
read from file -
Concurrency issues when using daskCache file conflicts, see #342 (closed) -
Fixes a bug where allow_overwrite
was ignored with thewrite.always
flag being set.
Can this MR be accepted?
-
Implementation ready -
Tests added or adjusted -
Documentation extended or updated
-
Code quality -
Changes follow coding guidelines -
Checked code coverage on new and adjusted code
-
-
Ready for merging -
Pipeline passes without warnings -
History cleaned-up or squash option set -
Changelog entry added -
Version number bumped
-
Related issues
Closes #342 (closed)
Edited by Yunus Sevinchan