Reconcile should support reconciling only a single repository
Right now praefect reconcile
reconciles the entire virtual storage when ran. This can be problematic when trying to fix data loss in a failover scenario where none of the nodes have the complete data but together do have all the data. Consider the following scenario:
- Primary:
gitaly-1
- Secondaries:
gitaly-2
,gitaly-3
gitaly-1
goes offline and is never seen again. New configuration:
- Primary:
gitaly-2
- Secondary:
gitaly-3
When the failover happened, some replication jobs were not successfully processed:
-
gitaly-2
is missing data fromrepo1
-
gitaly-3
is missing data fromrepo2
andrepo3
In order to resolve the error state, the administrator would like to reconcile repo1
from gitaly-3
to gitaly-2
. Right now, we do not support running reconcile on a per repository basis, meaning we would also overwrite gitaly-2
's repo2
and repo3
with the outdated ones from gitaly-3
@zj-gitlab this should be kept in mind when rolling failover to production. If the previous primary can be brought back online, one can always reconcile from there. If the previous primary is gone, then this will be a problem