E2E test of Praefect automatic reconciliation
Test automatic repository reconciliation
Docs: https://docs.gitlab.com/ee/administration/gitaly/recovery.html#automatic-reconciliation
Demo: gitlab-org/gitaly#3136 (closed)
Steps
- Enabled automatic reconciliation
praefect['reconciliation_scheduling_interval'] = '5s'
(ingitlab.rb
?). - Create a project.
- Stop one of the Gitaly nodes.
- Verify the Gitaly node is down (see prometheus data in the Grafana dashboards 'Virtual storage primary flapping').
- Push new data to the repository.
- Turn off the remaining Gitaly nodes.
- Bring back first Gitaly node.
- Check the repository is read-only.
- Check that the new data is missing.
- Try to write to the repository (it should fail).
- Bring a second Gitaly node back online.
- Check the logs for "msg":"reconciliation jobs scheduled".
- Check there's no dataloss anymore via the
dataloss
command. - Check that pushes succeed.