Improve Runner group release process - Part One - Automate GitLab Runner release steps
As a part of #29445 this is the first step I propose we implement in our road to something like this: #29445 (comment 1183598937).
Implementation
- Refer to the bottom of the issue for details of the problem and initial proposal that was presented *
In ggeorgiev_gitlab/gitlab-runner-group-releaser!1 (merged) I have created a pseudo project with pseudo code that has the skeleton of the first GitLab Runner releaser part it is as follows:
In order to release only GitLab Runner
- The release will be driven by a Go application. This will allow us to simplify the build scripts and pipelines. We would be able to run actions concurrently and synchronize the order in which they are executed.
- We would need a
prepare ci image
job to build the Go application in - The next job will be
release tags
. It would have to do the following actions inside the Go application - invoked withreleaser -create-tags $(GITLAB_RUNNER_RELEASE_TAGS)
:
# For each tag
* Important * Tags must be sorted and created in order from oldest to newest so they appear in the correct order in the /tags page of the project
All the checks can be done sequentially or out of order but the final tag creation must be done in order
1. git checkout main && git pull && git checkout -b 15-7-stable
2. make generate_changelog CHANGELOG_RELEASE=v15.7.0
3. Go through each changelog MR entry and verify that:
3.1 Milestone is correctly set to current milestone. If this is a patch release, e.g. 15.6.1 the milestone still needs to be the latest patch - %15.7
3.2 Correct labels are set: maintentance / feature / docs etc. Just check for their existance no need for further and deeper checks
3.3 Check the title doesn't contain prefixes - feat:, docs:
3.4 Check the title is properly capitalized so the final changelog is uniform
3.5 Try to collect as much info as possible, avoid failing fast
4. git add CHANGELOG.md && git commit -m "Update CHANGELOG for v15.7.0"
5. git tag v15.7.0 -m "Version v15.7.0" && git push origin v15.7.0 15-7-stable
Notes:
All the steps above should not break the workflow if they are ran twice. Changelog should check if it's generated, tag should be checked if it's created etc.
Finally:
Each tag should be merged into main
** ONLY ** if we are doing a normal .0 release - the main branch needs to have its version bumped to 15.8.0
The command bellow needs to be split into segments
git checkout main && \
git pull && \
git merge --no-ff 15-7-stable && \
echo -n "15.8.0" > VERSION && \
git add VERSION && \
git commit -m "Bump version to 15.8.0" && \
git push
- The next job will be
release wait tags
. It will be invoked withreleaser -wait-tags
and do the following:
# For each tag
Wait for the tag pipeline to pass if it fails let #f_runner_release know
No additional logic is needed for this iteration
- The next job will be
release to GitLab
. It will be invoked withreleaser -gitlab-release
and do the following:
# For each tag in order
We would need to rip the `stable gitlab release` job from the GitLab Runner project and figure out the commands from the
GitLab CLI to produce the relases at https:gitlab.com/gitlab-org/gitlab-runner/-/releases
It's also important that each release is issued in the correct order
- The final job would be
postrelease notify
-releaser -notify $(GITLAB_RUNNER_RELEASE_TAGS) $(GITLAB_RUNNER_RELEASE_CHANNEL)
Notify the #f_runner_release channel that the release is done, compile and provide relevant links, such as changelogs and release pages. The message could be formatted in a pretty way.
Problem
The biggest issue of GitLab Runner's release is the requirement for performing manual actions. Let's take for example all the steps for a GitLab Runner release (excluding Chart and other projects):
Runner release steps
-
Check if Pipeline for
main
is passing -
Pull latest changes
git checkout main && git pull && git checkout -b 15-6-stable
-
Generate changelog
make generate_changelog CHANGELOG_RELEASE=v15.6.0
-
Check changelog entries and commit
git add CHANGELOG.md && \ git commit -m "Update CHANGELOG for v15.6.0"
-
tag and push
git tag v15.6.0 -m "Version v15.6.0" && \ git push origin v15.6.0 15-6-stable
-
checkout to main, update VERSION file to 15.7.0 and push main:
git checkout main && \ git pull && \ git merge --no-ff 15-6-stable && \ echo -n "15.7.0" > VERSION && \ git add VERSION && \ git commit -m "Bump version to 15.7.0" && \ git push
Now, let's walk through them in details as if we are doing the release right now:
- Check if Pipeline for
main
is passing - Open https://gitlab.com/gitlab-org/gitlab-runner/-/commits/main and make sure the last commit that is not a docs commit (since the docs pipeline is shorter) passes. -
git checkout main && git pull && git checkout -b 15-6-stable
- Create a branch - simple enough -
make generate_changelog CHANGELOG_RELEASE=v15.6.0
- This might seem like a simple command but it has a lot to unpack. 1. It needs to downloadgitlab-changelog-generator
which if it has a new version you need to be aware of that as the Release manager. 2. You need to manually go through each MR that's generated in the newCHANGELOG.md
section and check that it has the correct title (e.g. no doc: prefix in the Docs section), check that it has the proper labels (MRs without the proper labels cannot be put in the proper category), check that if an MR has abug
label it's indeed a bug, check the milestone. -
Check changelog entries and commit
- Nothing much here but a commit - tag and push - just creating a tag
- At this point you need to monitor the pipeline and wait for its completion upon which you need to trigger a chart release since it requires runner images - that takes 2-3 hours
- If the pipeline has passed we can run
checkout to main, update VERSION file to 15.7.0 and push main
and merge the stable branch back to main.
At first this might not seem like a lot of work however:
- If we need to release a patch release with backports this work is tripled
- If we do a release and then a regression comes up this could explode to 6x times
Proposal
We start with our Umbrella project (Runner release project) detailed at #29445 (comment 1183598937).
This project for this first iteration should:
- Have a pipeline that is triggered manually
- The list of versions that will be released are passed as an environment variable.
- A dynamic child pipeline is generated for each version - https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines
- The tags need to be created in the correct order in the project. Maybe one job/pipeline for all tags would be better? Needs a bit of exploring but the general concept is the same
- (In a separate job?) a changelog is generated for each version and is sanity checked - do the MRs in the changelog fit all the criteria
- (In a separate job?) create a tag and push it
- (In a separate job?) monitor the tag pipeline and inform the release manager if it fails
- If it doesn't inform the release manager that it passed
- No need to trigger more pipelines for this issue - keep it into small iterations