chef-repo fails to apply changes when mirror has two commits
Problem
We use mirroring between https://gitlab.com/gitlab-com/gl-infra/chef-repo to push to https://ops.gitlab.net/gitlab-com/gl-infra/chef-repo, there is a limit to how frequent it can push, which is 5 minutes.
If we have 2 merge requests merged at around the same minute (which happens from time to time) we face the following problem:
- Both merge requests merged in https://gitlab.com/gitlab-com/gl-infra/chef-repo
MR-1
andMR-2
- Push mirror happens and pushes both merge commits at once for
MR-1
andMR-2
- Pipeline only starts on the last commit in this
MR-2
- Pipeline for
MR-1
never runs and the change is never applied to chef server - Engineer rolling out the change has to run
knife role from file $NAME
Some problems we are facing:
- Sometimes we miss this and we think we rolled out a change, but didn't
- Have to manually update a role locally
- It's not self-service and requires toil
Proposals
Upload all roles in every pipeline
This is because we look at the previous commit, one possibility is to look at all roles rather than the git diff.
The benefits we get from this:
- We make rule what's in
chef-repo
is always uploaded to chef-server - We don't have to worry about mirroring
One drawback of this might be that the pipeline might be slower since we have to do it for every role.
Have GitLab.com push to Ops.Gitlab.net
-
MR-1
andMR-2
are created on GitLab.com. - Both get merged close in time to each other.
- Each of them triggers a pipeline on the merge commit:
SHA-1
andSHA-2
. - These pipelines have a special job that
- pushes the
master
branch to ops mirror (as we need to have it there to work and mirroring will in most cases need few more minutes) - uses API to trigger a pipeline on ops mirror for the SHA of the merge.
- pushes the
- Ops mirror gets starts two pipelines:
-
pipeline-1
sourced fromSHA-1
which will handle changes fromMR-1
, -
pipeline-2
sourced fromSHA-2
which will handle changes fromMR-2
.
-
Edited by Steve Xuereb