Create new jobs to check that a CI config change doesn't break any use-case
I recently broke the forks workflow by changing a DAG configuration (#33862 (closed)).
We should have a CI job that would push the branch to different projects, with different branch names, create a WIP MR for them, and check the list of jobs created.
Proposal
Iteration 1 (stable branch)
Since jobs are created at the pipeline creation time, we could have a manual job (only on canonical MR) that would:
- Push the branch to canonical as a stable branch-matching name (no need to create a MR for these branches)
- Check the latest pipeline created for the new branch, and make sure that the jobs list matches the one of a reference stable branch pipeline.
Iteration 2 (auto-deploy branch)
Same as Iteration 1 but for auto-deploy branches.
Iteration 3 (normal fork)
- Same as Iteration 1 but push the branch as-is to a fork outside of
gitlab-org
- Open a MR from the fork to the canonical project
- Check the latest pipeline created for the new MR, and make sure that the jobs list matches the one of a reference stable branch pipeline.
Iteration 4 (security fork)
- Same as Iteration 1 but push the branch as-is to the security fork.
- Open a MR from the security fork to the security fork (or the canonical project?).
- Check the latest pipeline created for the new MR, and make sure that the jobs list matches the one of a reference stable branch pipeline.
master
)
Iteration 5 (canonical - Change the
rules
formaster
to also be enabled for^master-*
(we may need to double check some jobs, e.g. caches update etc.). - Same as Step 1. of iteration 1 but push the branch with as
master-<branch-name>
and change therules
formaster
to also be enabled for^master-*
(**we may need to double check some jobs, e.g. caches update etc.). - Same as Step 2. of iteration 1.
master
)
Iteration 6 (canonical FOSS - Same as Step 2. of iteration 5 but push the branch to canonical FOSS
master
. - Same as Step 3. of iteration 5.
Further iterations
- Dev
- Push branch with specific changes (e.g. code changes, QA changes, doc changes etc.) and open MRs against the original branch.
Resources
- https://about.gitlab.com/blog/2018/05/02/using-gitlab-ci-to-build-gitlab-faster/
- https://gitlab.com/gitlab-org/merge-train/blob/master/bin/merge-train
Additional discussions
Right now it's very easy to break GitLab's .gitlab-ci.yml (e.g. #34580 (closed) (closed)) because we don't test against possible branch names, especially with the needs and except keywords.
It'd be nice to have a CI step that does a check that the YAML is valid for various branch names. For example:
12-4-stable
12-4-stable-ee
12-4-auto-deploy-1234
- etc.
Maybe the easiest thing right now is to run a
bin/rails
runner script that does this?