Clear primitives during runtime
Fixes #313 (closed).
So currently during execution, we keep primitive instances in memory. This has few problems:
- They are kept between fit and produce phase. So if you do fit and produce at once, you might get a different result than if you do fit, pickle, unpickle, produce. (#316 (closed))
- Old primitives could store a lot of data (keep training data referenced on
self) and it is not possible to garbage collect that. - Because primitive stays in memory, its resources are never released. So it is not really possible to know when to release GPU resources (like GPU memory) so that another primitive in a pipeline might use them. Hopefully that happens when primitive is not executing, but there is really no way for a primitive to know that it is not needed anymore. With this MR primitive can install a hook to be called when the object is cleaned up.
Downsides:
- Primitive has really to store all its state (including random state) into
Paramsand useget_params/set_params. I think this is good and it should already be done, but it might break some primitives. - Calling
get_params/set_paramshas additional overhead than just keeping instances in memory. In combination with #443 (closed) it might be noticable.
Edited by Mitar