Schedule/record work prior to performing mutations
There are a few calls which perform a mutation and leave the Praefect cluster in a problematic state if replication jobs fail to be scheduled:
-
RenameRepositorymoves the repository on the disk directly. If the replication jobs fail to be scheduled, the client receives an error. Client can't retry the call afterwards, as the primary node would have already moved the repository. Since the replication jobs were not scheduled, the secondaries will remain in inconsistent state. I think the best course of action here would be to copy the repository instead of moving so the operation becomes idempotent. The client can then decide to retry if it receives an error or skip the operation. WillRenameRepositorystill be relevant after legacy storage is removed in %14.0? -
DeleteRepositorydeletes the repository directly on the primary and then schedules the replication jobs. If the jobs fail to be scheduled, the repository will be deleted on the primary but the secondaries remain in inconsistent state. The operation is idempotent and the inconsistency will be resolved if the client retries. If the client does not retry, the state remains inconsistent. One way to solve this would be to only schedule the repository for deletion when handling the RPC and to handle the deletion in the background later. This way the client gets an error only if we fail to persist the work. If we fail to persist the work, the repository stays consistent.
Similar problem is present with pushes for example, where we might update the primary but fail to persist the replication jobs. In such a case, if the client does not retry, we'll still consider the secondaries up to date and might serve different versions from the primary and the secondaries. This can be annoying and the client can solve this by pushing again. If the another client tries to push, it can be more difficult as the client needs to first fetch the changes possibly multiple times until it gets served the latest version from the primary.