Add proxy management, managing resources in a clever way
With the ProxyMixin.reinstate_proxy
method, it becomes possible to do some data management in scenarios where there is a lot of data / little memory.
Proposal
- Can use a
apply_ufunc
method inBaseDataGroup
, allowing to recursively apply thereinstate_proxy
method to certain leaves - Add a "Proxy Management" method the
DataManager
- Needs to be called explicitly
- Can use the
resource
module to determine whether memory is low - If a threshold is passed, can reinstate some proxies
- Ideally, reinstate those that were not used in a long time
- for that, could extend
.data
property of proxy mixin to store a timestamp
- for that, could extend
- Alternatively, have something like a global singular proxy registry, a
ProxyManager
, that takes care of all the stuff and is only invoked by theDataManager
...
Edited by Utopia Developers