Add Geo automated end-to-end replication tests
Problem
Geo doesn't have end-to-end replication tests, see #392980 (closed).
They could catch obvious, severe bugs, so we should add them.
Proposal
@nwestbury has begun developing a replication tester which can be used to benchmark replication latency in different deployment scenarios. This tester can be used to perform end-to-end QA testing as well, since it depends on working replication.
Alternatively, we could add more "GitLab QA" tests. For example, to avoid #392980 (closed), they could check if a registry record is created and moves to "synced" and "verified". Geo QA tests can already be run per branch, so this route has low barrier to entry.
-
Add 1 new QA test that creates a piece of data (like a project Git repo) on the primary site, and then queries the secondary site's
ProjectRepositoryRegistrys with the GraphQL API until it shows up and becomessyncedandverification succeeded. Allow for a very long timeout for now, e.g. 10 minutes. We can reduce it after gauging how the test behaves in CI and in the many other contexts in which QA tests are run. Weight 4. -
Open an issue for adding a similar QA test but for an LFS objectI tested LFS object replication in !224618 (merged) - Open an issue for adding a similar QA test but for a Container Repository => #593432
References
How to run Geo QA tests: Running Geo End to End QA Tests (yes we should migrate this into gitlab or gitlab-qa or gitlab-development-kit docs)