GitLab Runner 14.3 release checklist

Previous release checklist issue: #28106 (closed)

GitLab Runner Release manager: @tmaczukin

Deployment dashboard at: https://dashboards.gitlab.net/d/ci-runners-deployment/ci-runners-deployment-overview

First working day after 7th - v14.3.0-rc1 release

check if Pipeline for main is passing:
- add all required fixes to make main Pipeline passing
git checkout main && git pull in your local working copy!

prepare CHANGELOG entries

make generate_changelog CHANGELOG_RELEASE=v14.3.0-rc1

add v14.3.0-rc1 CHANGELOG entries and commit

git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v14.3.0-rc1" -S

tag and push v14.3.0-rc1:

git tag -s v14.3.0-rc1 -m "Version v14.3.0-rc1" && \
git push origin v14.3.0-rc1

create and push 14-3-stable branch:

git checkout -b 14-3-stable && \
git push -u origin 14-3-stable

checkout to main, update VERSION file to 14.4.0 and push main:

git checkout main; echo -n "14.4.0" > VERSION && \
git add VERSION && \
git commit -m "Bump version to 14.4.0" -S && \
git push

wait for Pipeline for v14.3.0-rc1 to pass
- add all required fixes to make v14.3.0-rc1 passing

update runner helm chart to use v14.3.0-rc1 version

check if Pipeline for main is passing:
- add all required fixes to make main Pipeline passing
go to your local working copy of https://gitlab.com/gitlab-org/charts/gitlab-runner
git checkout main && git pull in your local working copy!

set Helm Chart to use v14.3.0-rc1 version of GitLab Runner

sed -i".bak" "s/^appVersion: .*/appVersion: 14.3.0-rc1/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Update used GitLab Runner version to 14.3.0-rc1" -S

bump version of the Helm Chart to 0.33.0-rc1

sed -i".bak" "s/^version: .*/version: 0.33.0-rc1/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Bump version to 0.33.0-rc1" -S

prepare CHANGELOG entries
```
make generate_changelog CHANGELOG_RELEASE=v0.33.0-rc1
```
- manually add line including GitLab Runner version update information as the first item under New features
```
- Update GitLab Runner version to 14.3.0-rc1
```

add v0.33.0-rc1 CHANGELOG entries and commit

git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v0.33.0-rc1" -S

tag and push v0.33.0-rc1:

git tag -s v0.33.0-rc1 -m "Version v0.33.0-rc1" && \
git push origin v0.33.0-rc1

create and push 0-33-0-stable branch:

git checkout -b 0-33-0-stable && \
git push -u origin 0-33-0-stable

checkout to main, bump version of the Helm Chart to 0.34.0-beta, set back the bleeding version of Runner and push main:

git checkout main && \
sed -i".bak" "s/^version: .*/version: 0.34.0-beta/" Chart.yaml && \
rm Chart.yaml.bak && \
sed -i".bak" "s/^appVersion: .*/appVersion: bleeding/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Bump version to 0.34.0-beta" -S && \
git push

inform the g_runner channel that the feature freeze is done

❄ GitLab Runner v14.3.0-rc1 was tagged so the feature freeze is done! ❄

Everything merged to main as of this moment will be released with v14.4.0.
deploy v14.3.0-rc1

Detailed description of Runners deployment for GitLab.com CI fleet can be found in the runbook.
- at RC1 release day: to prmX runners
  - make sure it's not inside of the PCL time window.
  - go to your local chef-repo working directory and execute:
```
knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-prm-runners-to-14-3-0-rc1
```
  - update version
```
$EDITOR roles/gitlab-runner-prm.json
```
    In the role definition prepare the override_attributes entry. It should be placed at the top of the file:
```
"override_attributes": {
  "cookbook-gitlab-runner": {
    "gitlab-runner": {
        "repository": "unstable",
        "version": "14.3.0-rc1"
    }
  }
},
```
  - git add roles/gitlab-runner-prm.json && git commit -m "Update prmX runners to v14.3.0-rc1"
  - git push -u origin update-prm-runners-to-14-3-0-rc1 -o merge_request.create -o merge_request.label="deploy" -o merge_request.label="group::runner"
  - inform the EOC @sre-oncall inside of #production slack channel: @sre-oncall I'll be starting the deploy for v14.3.0-rc1 on prmX, as part of the monthly release. $LINK_TO_CHEF_REPO_MERGE_REQUEST
  - merge the chef-repo MR
  - check the production_dry_run job if it tries to update only the changed role
  - start the manual apply to prod job
  - after the job is finished execute:
```
knife ssh -C 1 -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh' &
time wait
```
- next day (if no problems): to rest of the runners
  - make sure it's not inside of the PCL time window.
  - go to your local chef-repo working directory and execute:
```
knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-runners-to-14-3-0-rc1
```
  - update version for gsrm/srm
```
$EDITOR roles/gitlab-runner-base.json
```
    In the role definition prepare the gitlab-runner entry:
```
"cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "unstable",
    "version": "14.3.0-rc1"
  }
}
```
  - update version for org-ci
```
$EDITOR roles/org-ci-base-runner.json
```
    In the role definition prepare the gitlab-runner entry:
```
"cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "unstable",
    "version": "14.3.0-rc1"
  }
}
```
  - remove overrides from prmX runners
```
$EDITOR roles/gitlab-runner-prm.json
```
  - git add roles/gitlab-runner-prm.json roles/gitlab-runner-base.json roles/org-ci-base-runner.json && git commit -m "Update runners to v14.3.0-rc1"
  - git push -u origin update-runners-to-14-3-0-rc1 -o merge_request.create -o merge_request.label="deploy" -o merge_request.label="group::runner"
  - inform the EOC @sre-oncall inside of #production slack channel: @sre-oncall I'll be starting the deploy for v14.3.0-rc1 on runner fleet, as part of the monthly release. $LINK_TO_CHEF_REPO_MERGE_REQUEST
  - merge the chef-repo MR
  - check the production_dry_run job if it tries to update only the changed role
  - start the manual apply to prod job
  - after the job is finished execute (we're not touching prmX - they are already updated):
```
knife ssh -C1 -afqdn 'roles:gitlab-runner-gsrm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:gitlab-runner-srm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:org-ci-base-runner' -- 'sudo  -i /root/runner_upgrade.sh' &
time wait
```

New features window is closed - things not merged into main up to this day, will be released with next release.

Pick into stable branch for RC2 phase

At this moment, until the next RC creation, we're in the pick into stable phase. The release manager should periodically review the list of merged MRs with a pick into X.Y label:

GitLab Runner MRs:

GitLab Runner's Helm Chart MRs:

For each such Merge Request the release manager should (reveal for details)

Find the merge commit for the Merge Request
Copy the commit reference
Checkout to the branch:
- git checkout 14-3-stable && git pull for GitLab Runner, or
- git checkout 0-33-0-stable && git pull for GitLab Runner's Helm Chart

Pick the merge commit with:

git cherry-pick -m 1 [merge commit SHA here]

Remove the Pick into X.Y label from the Merge Request and leave a note that it was picked:

for GitLab Runner use this message:

The MR was picked into `14-3-stable` branch and will be released with `v14.3.0-rc2`

/unlabel ~"Pick into 14.3"

for GitLab Runner's Helm Chart use this message:

The MR was picked into `0-33-0-stable` branch and will be released with `v0.33.0-rc2`

/unlabel ~"Pick into 0.33"

To prepare the `Pick into ...` MRs list execute:

rrhelper \
    list-pick-mrs-for-rc \
    --non-interactive \
    --do-not-create \
    --major 14 \
    --minor 3  \
    --helm-chart-major 0 \
    --helm-chart-minor 33 \
    28169 2

RC2 (if needed)

If any commit was picked into the stable branch since RC1 was released, RC2 version must be released no later than at 18th day of the month or immediately if changes were picked after 18th.

If you need to create RC2 version, then please execute this and follow the created checklist for RC2:

rrhelper \
    create-next-rc-checklist \
    --non-interactive \
    --do-not-create \
    --major 14 \
    --minor 3  \
    --helm-chart-major 0 \
    --helm-chart-minor 33 \
    28169 2

At 20th - the release day of GitLab Runner

check if Pipeline for 14-3-stable is passing:
- add all required fixes to make 14-3-stable Pipeline passing
git checkout 14-3-stable && git pull in your local working copy!
merge all RCx CHANGELOG entries into release entry

Also add all missing entries (should not happen at this moment unless an RC release was consciously skipped) and put a proper header at the beginning:
```
make generate_changelog CHANGELOG_RELEASE=v14.3.0
```
- merge all RC entries

add v14.3.0 CHANGELOG entries and commit

git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v14.3.0" -S

tag and push v14.3.0 and 14-3-stable:

git tag -s v14.3.0 -m "Version v14.3.0" && \
git push origin v14.3.0 14-3-stable

checkout to main and merge 14-3-stable into main, push main:

git checkout main && \
git pull && \
git merge --no-ff 14-3-stable
# check that the only changes are in CHANGELOG.md
git push

update runner helm chart to use v14.3.0 version
- check if Pipeline for 0-33-0-stable is passing:
  - add all required fixes to make 0-33-0-stable Pipeline passing
- go to your local working copy of https://gitlab.com/gitlab-org/charts/gitlab-runner
- git checkout 0-33-0-stable && git pull in your local working copy!
- update GitLab Runner version
```
sed -i".bak" "s/^appVersion: .*/appVersion: 14.3.0/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Update used GitLab Runner version to 14.3.0" -S
```
- bump version of the Helm Chart to 0.33.0
```
sed -i".bak" "s/^version: .*/version: 0.33.0/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Bump version to 0.33.0" -S
```
- merge all RCx CHANGELOG entries into release entry
  
  Also add all missing entries (should not happen at this moment unless an RC release was consciously skipped) and put a proper header at the beginning:
```
make generate_changelog CHANGELOG_RELEASE=v0.33.0
```
  - merge all RC entries
  - remove all Update GitLab Runner version to 14.3.0-rcX lines from the changes list
  - manually add line including GitLab Runner version update information as the first item under New features
```
- Update GitLab Runner version to 14.3.0
```
- add v0.33.0 CHANGELOG entries and commit
```
git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v0.33.0" -S
```
- tag and push v0.33.0 and 0-33-0-stable:
```
git tag -s v0.33.0 -m "Version v0.33.0" && \
git push origin v0.33.0 0-33-0-stable
```
- checkout to main and merge 0-33-0-stable into main (only this one time, to update CHANGELOG.md and make the tag available for ./scripts/prepare-changelog-entries.rb in next stable release), push main:
```
git checkout main && \
git pull && \
git merge --no-ff 0-33-0-stable
# check that the only changes are in CHANGELOG.md
git push
```

update Runner's chart version used by GitLab: gitlab!71559 (closed)

create branch

rrhelper \
    --dry-run \
    create-branch \
    gitlab-org/gitlab \
    update-gitlab-runner-helm-chart-to-0-33-0

create Merge Request

rrhelper \
    --dry-run \
    create-merge-request \
    --release-checklist-issue 28169 \
    --replace-link-in-release-checklist-issue link_to_MR__gce_rhcvu \
    --milestone 14.4 \
    --runner-version 14.3.0 \
    --helm-chart-version 0.33.0 \
    gitlab-org/gitlab \
    update-gitlab-runner-helm-chart-to-0-33-0 \
    master \
    runner-helm-chart-upgrade-in-gitlab \
    "Update GitLab Runner Helm Chart to 0.33.0/14.3.0"

Adjust and apply the patch to the GitLab CE sources

See the patch draft

Adjust the following patch (set proper current version; set the Merge Request ID in the CHANGELOG entry file) and save it at /tmp/patch.gitlab.

cat > /tmp/patch.gitlab << EOF
diff --git a/app/models/clusters/applications/runner.rb b/app/models/clusters/applications/runner.rb
index 0c0247da1fb..f17da0bb7b1 100644
--- a/app/models/clusters/applications/runner.rb
+++ b/app/models/clusters/applications/runner.rb
@@ -3,7 +3,7 @@
 module Clusters
   module Applications
     class Runner < ApplicationRecord
-      VERSION = '0.32.0'
+      VERSION = '0.33.0'

       self.table_name = 'clusters_applications_runners'
EOF

Go to your local GitLab CE working directory:

git checkout master && \
    git pull && \
    git checkout update-gitlab-runner-helm-chart-to-0-33-0 && \
    git apply /tmp/patch.gitlab && \
    git add . && \
    git commit -e -m "Update GitLab Runner Helm Chart to 0.33.0"

Add changelog trailer

Update GitLab Runner Helm Chart to 0.33.0

Changelog: other

Push changes

git push -u origin update-gitlab-runner-helm-chart-to-0-33-0

update Runner's chart version used by Cluster Management: gitlab-org/project-templates/cluster-management!22 (closed)

create branch

rrhelper \
    --dry-run \
    create-branch \
    gitlab-org/project-templates/cluster-management \
    update-gitlab-runner-helm-chart-to-0-33-0

create Merge Request

rrhelper \
    --dry-run \
    create-merge-request \
    --release-checklist-issue 28169 \
    --replace-link-in-release-checklist-issue link_to_MR__ghc_rhcvu \
    --milestone 14.3 \
    --runner-version 14.3.0 \
    --helm-chart-version 0.33.0 \
    gitlab-org/project-templates/cluster-management \
    update-gitlab-runner-helm-chart-to-0-33-0 \
    master \
    runner-helm-chart-upgrade-in-cluster-management \
    "feat: Update GitLab Runner Helm Chart to 0.33.0/14.3.0"

Adjust and apply the patch to the GitLab Helm Chart sources

See the patch draft

Adjust the following patch (set proper current version; set the Merge Request ID in the CHANGELOG entry file) and save it at /tmp/patch.gitlab-cluster-management.

cat > /tmp/patch.gitlab-cluster-management << EOF
diff --git a/applications/gitlab-runner/helmfile.yaml b/applications/gitlab-runner/helmfile.yaml
index 5b97c91..7799d50 100644
--- a/applications/gitlab-runner/helmfile.yaml
+++ b/applications/gitlab-runner/helmfile.yaml
@@ -6,7 +6,7 @@ releases:
 - name: runner
   namespace: gitlab-managed-apps
   chart: gitlab/gitlab-runner
-  version: 0.32.0
+  version: 0.33.0
   installed: true
   values:
     - values.yaml.gotmpl
EOF

Go to your local GitLab Cluster Application working directory:

git checkout master && \
    git pull && \
    git checkout update-gitlab-runner-helm-chart-to-0-33-0 && \
    git apply /tmp/patch.gitlab-cluster-management && \
    git add . && \
    git commit -m "feat: Update GitLab Runner Helm Chart to 0.33.0"

Push changes

git push -u origin update-gitlab-runner-helm-chart-to-0-33-0

At 22nd - the official day of GitLab release

wait for Pipeline for v14.3.0 to pass
- add all required fixes to make v14.3.0 passing
deploy v14.3.0

Detailed description of Runners deployment for GitLab.com CI fleet can be found in the runbook.
- at release day: to prmX runners
  - make sure it's not inside of the PCL time window.
  - go to your local chef-repo working directory and execute:
```
knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-prm-runners-to-14-3-0
```
  - update version
```
$EDITOR roles/gitlab-runner-prm.json
```
    In the role definition prepare the override_attributes entry. It should be placed at the top of the file:
```
"override_attributes": {
  "cookbook-gitlab-runner": {
    "gitlab-runner": {
        "repository": "gitlab-runner",
        "version": "14.3.0"
    }
  }
},
```
  - git add roles/gitlab-runner-prm.json && git commit -m "Update prmX runners to v14.3.0"
  - git push -u origin update-prm-runners-to-14-3-0 -o merge_request.create -o merge_request.label="deploy" -o merge_request.label="group::runner"
  - inform the EOC @sre-oncall inside of #production slack channel: @sre-oncall I'll be starting the deploy for v14.3.0 on prmX, as part of the monthly release. $LINK_TO_CHEF_REPO_MERGE_REQUEST
  - merge the chef-repo MR
  - check the production_dry_run job if it tries to update only the changed role
  - start the manual apply to prod job
  - after the job is finished execute:
```
knife ssh -C 1 -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh' &
time wait
```
- next day (if no problems): to rest of the runners
  - make sure it's not inside of the PCL time window.
  - go to your local chef-repo working directory and execute:
```
knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-runners-to-14-3-0
```
  - update version for gsrm/srm
```
$EDITOR roles/gitlab-runner-base.json
```
    In the role definition prepare the gitlab-runner entry:
```
"cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "gitlab-runner",
    "version": "14.3.0"
  }
}
```
  - update version for org-ci
```
$EDITOR roles/org-ci-base-runner.json
```
    In the role definition prepare the gitlab-runner entry:
```
"cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "gitlab-runner",
    "version": "14.3.0"
  }
}
```
  - remove overrides from prmX runners
```
$EDITOR roles/gitlab-runner-prm.json
```
  - git add roles/gitlab-runner-prm.json roles/gitlab-runner-base.json roles/org-ci-base-runner.json && git commit -m "Update runners to v14.3.0"
  - git push -u origin update-runners-to-14-3-0 -o merge_request.create -o merge_request.label="deploy" -o merge_request.label="group::runner"
  - inform the EOC @sre-oncall inside of #production slack channel: @sre-oncall I'll be starting the deploy for v14.3.0 on runner fleet, as part of the monthly release. $LINK_TO_CHEF_REPO_MERGE_REQUEST
  - merge the chef-repo MR
  - check the production_dry_run job if it tries to update only the changed role
  - start the manual apply to prod job
  - after the job is finished execute (we're not touching prmX - they are already updated):
```
knife ssh -C1 -afqdn 'roles:gitlab-runner-gsrm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:gitlab-runner-srm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:org-ci-base-runner' -- 'sudo -i /root/runner_upgrade.sh' &
time wait
```

Before next 7th

chose a release manager

create a Pick into 0.34 label in Helm Chart's project

rrhelper \
    --dry-run \
    create-label \
    6329679 \
    "Pick into 0.34" \
    "Merge Requests to pick into 0-34-0 stable branch" \
    "#00C8CA"

create the new Release Checklist issue: #28242 (closed)

rrhelper \
    create-release-checklist \
    --do-not-create \
    --non-interactive \
    --replace-link-in-previous-release-checklist-issue link_to_new_checklist_issue \
    --release-manager-handle release_manager \
    --previous-release-checklist-issue 28169

Edited Oct 01, 2021 by Tomasz Maczukin