Implementation suggestions for HyperParameters class
I have implementation suggestions for HyperParameters class.
The HyperParameters.__add__ works in a way where both of the operands are being copied and a new HyperParameters object is returned.
However, for in place addition as in param1 += param2, the object.__iadd__ method should be used.
In this case, only param2 will need to be deep copied and the param1 can be updated accordingly.
This should be a much elegant and more memory saving operation.
I don't know if this is a good thing or not, but since deep copying can be expensive (what if param2 has network weight already loaded?),
maybe we can add a toggle variable in HyperParameters like HyperParameters.__deepcopy.
For example, param1 can check param2.deepcopy (maybe @property can be used to access HyperParameters.__deepcopy)
during the __iadd__ operation to determine weather the variables in param2 should be deep copied or not.
By default, HyperParameters.__deepcopy will be set to True, but power users that know what they are doing
can toggle it maybe by calling HyperParameters.disable_deepcopy.