Making large structs immutable
The cogs utilize multiple structs which grow with the size of experiments. The issue is that many of these structs have members which need to be updated (say an Action's Complete field for a Task struct).
The problem encountered with this methodology is that these updates are written back to etcd which has constraints on the size of requests (https://github.com/etcd-io/etcd/blob/master/Documentation/dev-guide/limit.md). So as our materializations grow in the size, the ability to write them to etcd is limited.
The solution should be in creating large immutable objects (such as mzinfo, or task) which do not change, but include pointers to the much smaller structs that do change (as an example NodeState).