Geo: improve snapshot approach to preserve existing copy on secondary in case primary is corrupt
Context
If the primary node gets corrupted for some reason, doing a git fetch against it will not spread out some corruption cases (like missing objects). This is the method we use to synchronize the secondary against the primary.
There is a catch here: in the secondary we have a fallback mechanism that switches to transfering a tarball from primary to the secondary when the secondary fails to sync it for several times. This was intended to be a failsafe mechanism for example, when git is crashing on the otherside due to it using too much memory when trying to transfer a huge repository.
Because of that fallback mechanism there is a chance that a corrupted repository from the primary will replace a perfectly fine repository on the secondary when that mechanism is activated.
Proposal
With the intention of preserving data, we should consider a different approach here.
Instead of always replacing existing repositories with tarball (when that mechanism is activated), we should quarantine existing repository (by issuing a tarball against it self) on the secondary, before replacing with the one coming from the primary.
By doing that we allow any future attempt to restore data to be possible.
cc @fzimmer