Fingerprint Bucket Rotation

MERGE REQUEST

MR Guidelines

Overview

This MR aims to add several layers of hardening to the process of fingerprint bucket rotation. There is a case in production where this fails, and where this host is stuck with an account manager that has withdrawals deactivated. I can only guess at the true cause because I was unable to reproduce it locally, however it most probably was caused either by a corruption of the file (or filehandle) on the host in question, or a race condition where callProcessConsensusChange was called twice in rapid succession.

In short:

• I ensured syncAndClose happens under lock (even though this was not the source of the issue)
• I ensured errors during syncAndClose do not prevent reopening the buckets, this is very important as it ensures we can recover from a bad state, and not stay in a state of deadlock forever
• I ensured the removal of old fingerprint buckets actually only removes old buckets. The way it was implemented there was a race condition where it might remove a recently opened bucket file.

I believe that last bullet is the cause of the linked issue but was unable to reproduce.

Example for Visual Changes

N/A

Checklist

Review and complete the checklist to ensure that the MR is complete before assigned to an approver.

• All new methods or updated methods have clear docstrings
• Testing added or updated for new methods
• Any new packages are added to Makefile and .gitlab-ci.yml
• API documentation updated for API updates
• Module README.md updated for changes to workflow
• Issue added to Sia-UI repo for new supporting features
• Changelog File Created

Issues Closed

Closes #4211 (closed)