Fault tolerance in the producer consumer model

The Cogs system is designed around a producer consumer model where the driver receives commands from Merge, produces tasks and then cogs consume tasks. However, when things go sideways - the relevant state does not always get cleaned up. We should think about what the fault model is in this producer consumer system, and how to ensure that faults that occur do not leave behind state that will interfere with future materializations.