Make CI clones more robust in the face of replication lag
In https://gitlab.com/gitlab-com/infrastructure/issues/4125, we saw that increased replication lag caused CI builds to fail, most likely due to the fact that the build token hadn't been propagated to the secondaries.
Is there a way we can be more robust here? Some ideas:
- Always go to the primary to check the build token: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5958
- If replication lag > amount and build token is invalid, check the primary: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5960
/cc: @ayufan, @bikebilly
Edited by Kamil Trzciński