Make replication tests check registry data

Problem

Most or all Geo replication tests are structured like:

Setup: Create a resource
Check: Call an API endpoint on the secondary site to check existence

This procedure was valid when secondary sites were all read-only, and data would appear to not exist until it became replicated.

Now, browsing a secondary site or making API requests against it will be forwarded to the primary. Additionally, Git requests are forwarded as soon as the secondary site becomes aware that a repo has changed. Therefore the resource will immediately appear to exist even if it is not replicated yet.

Therefore the replication tests currently test proxying behavior.

Proposal

The Check step must be done in such a way that it fails when replication is not complete, and succeeds when replication is complete.

With the SSF, one way to perform the Check step with 99% confidence is to poll a GraphQL query (which already exists and is in-use by the UI for Admin > Geo > Replicable Details) for the registry records, and observe the state field move from pending or started to synced.

We can get 99.5% confidence by additionally waiting for the registry record's verification_state field to move to verification_succeeded.

These numbers are made up, but note that confidence can never be 100%. For example, the code can have a bug which lies to you.

For example, you can ask for LFS Object registry data of a specific Geo node like so:

➜  geo-replication-tester git:(mk/base-ssl-on-url-schema) curl -H "PRIVATE-TOKEN: $GITLAB_PAT" https://gdk2.test:3444/api/v4/geo/node_proxy/2/graphql -X POST -i -H 'Content-Type: application/json' --data "{\"query\": \"query { geoNode { lfsObjectRegistries(first: 100, last: null) { pageInfo { ...PageInfo __typename } nodes { id state retryCount lastSyncFailure retryAt lastSyncedAt verifiedAt @include(if: true) createdAt __typename } __typename } __typename } } fragment PageInfo on PageInfo { hasNextPage hasPreviousPage startCursor endCursor __typename }\"}"
HTTP/2 200 
server: nginx/1.23.3
date: Thu, 23 Feb 2023 20:50:24 GMT
content-type: application/json
cache-control: max-age=0, private, must-revalidate
content-security-policy: default-src 'none'
etag: W/"e10d279747e3a6f4eaea65f4abc6bedd"
vary: Origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-request-id: 01GT00N79WCD6Y4TRKYPV82DB4
x-runtime: 0.161362

{"data":{"geoNode":{"lfsObjectRegistries":{"pageInfo":{"hasNextPage":false,"hasPreviousPage":false,"startCursor":"eyJpZCI6IjEifQ","endCursor":"eyJpZCI6IjI5In0","__typename":"PageInfo"},"nodes":[{"id":"gid://gitlab/Geo::LfsObjectRegistry/1","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-06-14T20:01:31Z","verifiedAt":"2023-02-06T18:58:31Z","createdAt":"2022-06-14T19:59:55Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/2","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-06-14T20:01:37Z","verifiedAt":"2023-02-06T18:58:31Z","createdAt":"2022-06-14T19:59:55Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/3","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-06-14T20:01:52Z","verifiedAt":"2023-02-06T18:58:31Z","createdAt":"2022-06-14T19:59:55Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/4","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-06-14T20:01:53Z","verifiedAt":"2023-02-06T18:58:31Z","createdAt":"2022-06-14T19:59:55Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/5","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-06-14T20:01:55Z","verifiedAt":"2023-02-06T18:58:32Z","createdAt":"2022-06-14T19:59:55Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/6","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:19Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/7","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:25Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/8","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:27Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/9","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:32Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/10","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:34Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/11","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:37Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/12","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:39Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/13","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:41Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/14","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:42Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/15","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:44Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/16","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:45Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/17","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:47Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/18","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:48Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/19","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:50Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/20","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:51Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/21","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:53Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/22","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:54Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/23","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-09-09T20:13:56Z","verifiedAt":null,"createdAt":"2022-09-03T00:35:13Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/24","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-10-25T00:02:29Z","verifiedAt":null,"createdAt":"2022-10-25T00:02:29Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/25","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-10-25T00:08:03Z","verifiedAt":null,"createdAt":"2022-10-25T00:08:03Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/26","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-10-25T00:08:03Z","verifiedAt":null,"createdAt":"2022-10-25T00:08:03Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/27","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-10-25T00:08:03Z","verifiedAt":null,"createdAt":"2022-10-25T00:08:03Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/28","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-10-25T00:08:03Z","verifiedAt":null,"createdAt":"2022-10-25T00:08:03Z","__typename":"LfsObjectRegistry"},{"id":"gid://gitlab/Geo::LfsObjectRegistry/29","state":"SYNCED","retryCount":0,"lastSyncFailure":null,"retryAt":null,"lastSyncedAt":"2022-10-25T00:08:03Z","verifiedAt":null,"createdAt":"2022-10-25T00:08:03Z","__typename":"LfsObjectRegistry"}],"__typename":"LfsObjectRegistryConnection"},"__typename":"GeoNode"}}}%

Edited Feb 23, 2023 by Michael Kozono