Hash tables for assembling sparse matrix without needing preallocation
We have several use cases in MOOSE where it is very convenient to use function/residual information from early stages of its evaluation in later stages of its evaluation. I will outline one such case. We have a "kinematic" mechanical contact formulation in which the computed traction forces from the secondary side (evaluated with our "Kernel" objects early in the function evaluation) are moved over to the primary side and then we apply a constraint on the secondary side. These forces are the kernel residuals. The easiest way to get these forces in the lately evaluated contact objects is to read from the residual vector. Consequently we must make sure that the residual (Jacobian) is assembled before we attempt to read in order to ensure that we have the complete summation of traction forces. However, in the case of Jacobian evaluation, assembly deletes the memory allocation for the portions of the matrix representing the coupling between the disconnected secondary and primary surfaces. So then when we go to transfer the data from the secondary side to the primary side we have to perform new matrix memory allocations which is very slow (or results in an error depending on what we've set for MAT_NEW_NONZERO_ALLOCATION_ERR
).
There are work-arounds for this. @fdkong implemented the reset matrix preallocation routines which we currently use. However, this only works if our original sparsity pattern sufficiently covers the dof couplings that will occur over the course of the simulation/nonlinear solve. In order to sufficiently cover possible dof couplings for these moving mesh contact problems we commonly have to overspecify our sparsity pattern/matrix preallocation, e.g. maybe the secondary mesh will move relative to the primary mesh west one element or maybe east three elements or maybe north 5 elements, so to be safe we have to preallocate possible nonzeros for all those scenarios. This leads to overallocation and bad data locality.
@fdkong has talked to me about an alternative assembly implementation using hash tables in which the actual matrix object is only assembled once at the very end of Jacobian evaluation/before we setup the preconditioner, and any intermediate data required is read from the hash table data structures. It sounds like with this implementation we could do whatever we want in our residual/Jacobian functions and not worry so much about our preallocation.