Come up with a uniform, consistent strategy for handing mirrors to ops.gitlab.net
The process for mirroring repositories to ops.gitlab.net is case-by-case. In some cases gitlab.com is the source of truth and for others it is not. This issue is to come up with a sensible consistent plan for repository mirroring between ops.gitlab.net and gitlab.com.
Option 1: Source issues and MRs and code changes on gitlab.com, sync repos to ops.gitlab.net with pipelines on ops.gitlab.net
The following repositories are affected:
- chef-repo (private)
- gitlab-com-infrastructure (public)
- deploy-tooling (public)
- deployer (public)
- terraform-modules (public)
We would leave patcher on ops.gitlab.net since it is the most likely project to use in an outage scenario
Pros
- our workflows are more transparent, MRs and comments on MRs are on gitlab.com
- if gitlab.com is unavailable we can still run pipelines since runners are connected to the ops instance
- issues can be linked to MRs
Cons
- It's more difficult to dogfood gitlab.com pipeline features when we have split between ops and gitlab.com
- It's possible for pipelines to fail after the sync to ops.gitlab.net which can be delayed
- If pipelines run on ops with multiple stages, they are detached from the MR review
- Any pipelines that need to run on gitlab.com as well must have duplicated CI variable config
- If GitLab.com is unavailable we need to completely change our workflow, update remotes, and use a different instance
Option: 2: Source issues on gitlab.com, MRs and code changes on ops.gitlab.net, sync repos to gitlab.com with pipelines on ops.gitlab.net
Pros
- It's better for dogfooding GitLab devops features, it's more like what the customer would run where MRs are submitted to the same instance as where ops pipelines run
- If gitlab.com is unavailable we can still run pipelines since runners are connected to the ops instance
- If gitlab.com is unavailable we can still commit our code normally without having to update remotes
- The community can still our code since it is mirrored to gitlab.com
Cons
- Issues cannot be linked to MRs except by adding links to the other instance, see this example where MRs are not accessible to the public because they are linked to an internal ops project.
- Workflows are not as transparent as the community cannot view MRs and comments as easily. In order for the community to view MRs the corresponding projects need to be public on ops.
Option 3, a combination of both
(this is not a good option but is what we have currently)
repositories for infrastructure
| repository | source | mirror | pipelines that run for publishing/deploying |
|---|---|---|---|
| gitlab-cookbooks/chef-repo | ops.gitlab.net | none | ops.gitlab.net |
| gitlab-com/gitlab-com-infrastructure | ops.gitlab.net | gitlab.com | ops.gitlab.net |
| gitlab-com/gl-infra/terraform-modules | ops.gitlab.net | gitlab.com | ops.gitlab.net |
| gitlab-cookbooks/* (except chef-repo) | gitlab.com | ops.gitlab.net (pull mirror) | ops.gitlab.net |
| gitlab-com/runbooks | gitlab.com | dev.gitlab.org | none |
| gitlab-com/gl-infra/deployer | ops.gitlab.net | gitlab.com | ops.gitlab.net |
| gitlab-com/gl-infra/patcher | ops.gitlab.net | gitlab.com | ops.gitlab.net |
| gitlab-com/chatops | gitlab.com | ops.gitlab.net | ops.gitlab.net |
Proposals
- make everything under
gitlab-cookbooks/*sourced on ops.gitlab.net? - come up with a script that makes it easier to add new repos to this list?
- should we simply mirror everything under
gitlab-com,gl-infra, andgitlab-org? - add new groups so that there is one group for gitlab.com sourcing, and another group for ops.gitlab.net sourcing?
Edited by John Jarvis