Skip to content

Address concurrency and cache-reading issues with forced cache read/write

Yunus Sevinchan requested to merge fix-mp-cache-issues into main

This MR aims to address issues raised by @peanutfun in #342 (closed).

  • Allow not to unpack loaded data cache objects (i.e. return cont.data instead of the cont the data was loaded into).
    • This is not a long-term solution, but a temporary workaround to customise whether loaded objects should again be dantro containers. In the long run, this requires more intelligent saving of cache files that allow restoring into exactly the type that they were before.
      read:
        enabled: true
        always: true
        load_options:
          unpack: false
          chunks: {...}
          # ... further options
  • Fixes a bug where the DataManager acted as an unwanted cache layer despite being configured to always read from file
  • Concurrency issues when using dask Cache file conflicts, see #342 (closed)
  • Fixes a bug where allow_overwrite was ignored with the write.always flag being set.

Can this MR be accepted?

  • Implementation ready
  • Tests added or adjusted
  • Documentation extended or updated
  • Code quality
  • Ready for merging
    • Pipeline passes without warnings
    • History cleaned-up or squash option set
    • Changelog entry added
    • Version number bumped

Related issues

Closes #342 (closed)

Edited by Yunus Sevinchan

Merge request reports