Skip to content

Add proxy management, managing resources in a clever way

With the ProxyMixin.reinstate_proxy method, it becomes possible to do some data management in scenarios where there is a lot of data / little memory.

Proposal

  • Can use a apply_ufunc method in BaseDataGroup, allowing to recursively apply the reinstate_proxy method to certain leaves
  • Add a "Proxy Management" method the DataManager
    • Needs to be called explicitly
    • Can use the resource module to determine whether memory is low
    • If a threshold is passed, can reinstate some proxies
    • Ideally, reinstate those that were not used in a long time
      • for that, could extend .data property of proxy mixin to store a timestamp
  • Alternatively, have something like a global singular proxy registry, a ProxyManager, that takes care of all the stuff and is only invoked by the DataManager...
Edited by Utopia Developers
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information