Handle RemoveRepository in Praefect
Repository deletions are currently problematic in Praefect. Removals are first performed on the Gitaly nodes and afterwards the repository's database records are updated. If the process fails, it could be that the repository is removed from all or some of the Gitalys without updating the database state. This can cause a weird in-between state where Praefect believes repositories to exist when they actually don't. The removals are also handled as regular mutators. This means it's not possible to delete a repository which doesn't have any up to date replicas available. This doesn't make sense given whether or not the replicas are up to date does not matter when the repository is being removed. Likewise, they are currently handled transactionally to workaround avoid the races between recreating the repository and removing all of the state prior to the recreation. Handling deletions transactionally and verifying all of the Gitalys perform the same operation is also not necessary as we'll anyway end up removing the repository.
Better way would be to handle the removal in the database and afterwards clean up the disks properly. This way Praefect doesn't end up in a state where we have records for repositories which should not exist anymore. Once the database records are removed, Praefect attempts to clean up the replicas in known locations. It's still possible that this clean up fails or that there are replicas on some of the storages which did not have database records. This is a better scenario, as it's easier for the administrator to remove the extra replicas from the disks than to modify the database state. Praefect will periodically walk the storages to identify any stale repositories. This walking would also identify these leftovers and eventually lead to their removal. Once Praefect begins generating unique paths for repositories, these stale repositories won't even prevent the creation of a new repository with the same virtual storage and relative path.
Related to: #3485 (closed)