Skip to content

GitLab Runner 13.7 release checklist

Previous release checklist issue: #27202 (closed)

GitLab Runner Release manager: @tmaczukin

First working day after 7th - v13.7.0-rc1 release

  • check if Pipeline for master is passing: pipeline status

    • add all required fixes to make master Pipeline passing

  • git checkout master && git pull in your local working copy!

  • prepare CHANGELOG entries

    make generate_changelog CHANGELOG_RELEASE=v13.7.0-rc1

  • add v13.7.0-rc1 CHANGELOG entries and commit

    git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v13.7.0-rc1" -S

  • tag and push v13.7.0-rc1:

    git tag -s v13.7.0-rc1 -m "Version v13.7.0-rc1" && \
git push origin v13.7.0-rc1

  • create and push 13-7-stable branch:

    git checkout -b 13-7-stable && \
git push -u origin 13-7-stable

  • checkout to master, update VERSION file to 13.8.0 and push master:

    git checkout master; echo -n "13.8.0" > VERSION && \
git add VERSION && \
git commit -m "Bump version to 13.8.0" -S && \
git push

  • wait for Pipeline for v13.7.0-rc1 to pass pipeline status

    • add all required fixes to make v13.7.0-rc1 passing

  • update runner helm chart to use v13.7.0-rc1 version

    • check if Pipeline for master is passing: pipeline status

      • add all required fixes to make master Pipeline passing

    • go to your local working copy of https://gitlab.com/gitlab-org/charts/gitlab-runner

    • git checkout master && git pull in your local working copy!

    • set Helm Chart to use v13.7.0-rc1 version of GitLab Runner

      sed -i".bak" "s/^appVersion: .*/appVersion: 13.7.0-rc1/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Update used GitLab Runner version to 13.7.0-rc1" -S

    • bump version of the Helm Chart to 0.24.0-rc1

      sed -i".bak" "s/^version: .*/version: 0.24.0-rc1/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Bump version to 0.24.0-rc1" -S

    • prepare CHANGELOG entries

      make generate_changelog CHANGELOG_RELEASE=v0.24.0-rc1

      • manually add line including GitLab Runner version update information as the first item under New features

        - Update GitLab Runner version to 13.7.0-rc1

    • add v0.24.0-rc1 CHANGELOG entries and commit

      git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v0.24.0-rc1" -S

    • tag and push v0.24.0-rc1:

      git tag -s v0.24.0-rc1 -m "Version v0.24.0-rc1" && \
git push origin v0.24.0-rc1

    • create and push 0-24-0-stable branch:

      git checkout -b 0-24-0-stable && \
git push -u origin 0-24-0-stable

    • checkout to master, bump version of the Helm Chart to 0.25.0-beta, set back the bleeding version of Runner and push master:

      git checkout master && \
sed -i".bak" "s/^version: .*/version: 0.25.0-beta/" Chart.yaml && \
rm Chart.yaml.bak && \
sed -i".bak" "s/^appVersion: .*/appVersion: bleeding/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Bump version to 0.25.0-beta" -S && \
git push

  • inform the g_runner channel that the feature freeze is done

    ❄ GitLab Runner v13.7.0-rc1 was tagged so the feature freeze is done! ❄

    Everything merged to master as of this moment will be released with v13.8.0.

  • deploy v13.7.0-rc1

    Detailed description of Runners deployment for GitLab.com CI fleet can be found in the runbook.

    • at RC1 release day: to prmX runners

      • go to your local chef-repo working directory and execute:

        knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-prm-runners-to-13-7-0-rc1

      • update version

        $EDITOR roles/gitlab-runner-prm.json

        In the role definition prepare the override_attributes entry. It should be placed at the top of the file:

        "override_attributes": {
  "cookbook-gitlab-runner": {
    "gitlab-runner": {
        "repository": "unstable",
        "version": "13.7.0-rc1"
    }
  }
},

      • git add roles/gitlab-runner-prm.json && git commit -m "Update prmX runners to v13.7.0-rc1"

      • git push -u origin update-prm-runners-to-13-7-0-rc1

      • after pushing the branch, create and manage to merge the chef-repo MR

      • check the production_dry_run job if it tries to update only the changed role

      • start the manual apply to prod job

      • after the job is finished execute:

        knife ssh -C 1 -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh' &
time wait

    • next day (if no problems): to rest of the runners

      • go to your local chef-repo working directory and execute:

        knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-runners-to-13-7-0-rc1

      • update version for gsrm/srm

        $EDITOR roles/gitlab-runner-base.json

        In the role definition prepare the gitlab-runner entry:

        "cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "unstable",
    "version": "13.7.0-rc1"
  }
}

      • update version for org-ci

        $EDITOR roles/org-ci-base-runner.json

        In the role definition prepare the gitlab-runner entry:

        "cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "unstable",
    "version": "13.7.0-rc1"
  }
}

      • remove overrides from prmX runners

        $EDITOR roles/gitlab-runner-prm.json

      • git add roles/gitlab-runner-prm.json roles/gitlab-runner-base.json roles/org-ci-base-runner.json && git commit -m "Update runners to v13.7.0-rc1"

      • git push -u origin update-runners-to-13-7-0-rc1

      • after pushing the branch, create and manage to merge the chef-repo MR

      • check the production_dry_run job if it tries to update only the changed role

      • start the manual apply to prod job

      • after the job is finished execute (we're not touching prmX - they are already updated):

        knife ssh -C1 -afqdn 'roles:gitlab-runner-gsrm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:gitlab-runner-srm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:org-ci-base-runner' -- 'sudo  -i /root/runner_upgrade.sh' &
time wait

New features window is closed - things not merged into master up to this day, will be released with next release.

Pick into stable branch for RC2 phase

At this moment, until the next RC creation, we're in the pick into stable phase. The release manager should periodically review the list of merged MRs with a pick into X.Y label:

GitLab Runner MRs:

GitLab Runner's Helm Chart MRs:

For each such Merge Request the release manager should (reveal for details)

  1. Find the merge commit for the Merge Request

  2. Copy the commit reference

  3. Checkout to the branch:

    • git checkout 13-7-stable && git pull for GitLab Runner, or
    • git checkout 0-24-0-stable && git pull for GitLab Runner's Helm Chart

  4. Pick the merge commit with:

    git cherry-pick -m 1 [merge commit SHA here]

  5. Remove the Pick into X.Y label from the Merge Request and leave a note that it was picked:

  • for GitLab Runner use this message:

    The MR was picked into `13-7-stable` branch and will be released with `v13.7.0-rc2`

/unlabel ~"Pick into 13.7"

  • for GitLab Runner's Helm Chart use this message:

    The MR was picked into `0-24-0-stable` branch and will be released with `v0.24.0-rc2`

/unlabel ~"Pick into 0.24"
To prepare the `Pick into ...` MRs list execute: 
rrhelper \
    list-pick-mrs-for-rc \
    --non-interactive \
    --do-not-create \
    --major 13 \
    --minor 7  \
    --helm-chart-major 0 \
    --helm-chart-minor 24 \
    27282 2

RC2 (if needed)

If any commit was picked into the stable branch since RC1 was released, RC2 version must be released no later than at 18th day of the month or immediately if changes were picked after 18th.

If you need to create RC2 version, then please execute this and follow the created checklist for RC2:

rrhelper \
    create-next-rc-checklist \
    --non-interactive \
    --do-not-create \
    --major 13 \
    --minor 7  \
    --helm-chart-major 0 \
    --helm-chart-minor 24 \
    27282 2

At 20th - the release day of GitLab Runner

  • check if Pipeline for 13-7-stable is passing: pipeline status

    • add all required fixes to make 13-7-stable Pipeline passing

  • git checkout 13-7-stable && git pull in your local working copy!

  • merge all RCx CHANGELOG entries into release entry

    Also add all missing entries (should not happen at this moment unless an RC release was consciously skipped) and put a proper header at the beginning:

    make generate_changelog CHANGELOG_RELEASE=v13.7.0
    • merge all RC entries

  • add v13.7.0 CHANGELOG entries and commit

    git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v13.7.0" -S

  • tag and push v13.7.0 and 13-7-stable:

    git tag -s v13.7.0 -m "Version v13.7.0" && \
git push origin v13.7.0 13-7-stable

  • checkout to master and merge 13-7-stable into master, push master:

    git checkout master && \
git pull && \
git merge --no-ff 13-7-stable
# check that the only changes are in CHANGELOG.md
git push

  • update runner helm chart to use v13.7.0 version

    • check if Pipeline for 0-24-0-stable is passing: pipeline status

      • add all required fixes to make 0-24-0-stable Pipeline passing

    • go to your local working copy of https://gitlab.com/gitlab-org/charts/gitlab-runner

    • git checkout 0-24-0-stable && git pull in your local working copy!

    • update GitLab Runner version

      sed -i".bak" "s/^appVersion: .*/appVersion: 13.7.0/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Update used GitLab Runner version to 13.7.0" -S

    • bump version of the Helm Chart to 0.24.0

      sed -i".bak" "s/^version: .*/version: 0.24.0/" Chart.yaml && \
rm Chart.yaml.bak && \
git add Chart.yaml && \
git commit -m "Bump version to 0.24.0" -S

    • merge all RCx CHANGELOG entries into release entry

      Also add all missing entries (should not happen at this moment unless an RC release was consciously skipped) and put a proper header at the beginning:

      make generate_changelog CHANGELOG_RELEASE=v0.24.0

      • merge all RC entries

      • remove all Update GitLab Runner version to 13.7.0-rcX lines from the changes list

      • manually add line including GitLab Runner version update information as the first item under New features

        - Update GitLab Runner version to 13.7.0

    • add v0.24.0 CHANGELOG entries and commit

      git add CHANGELOG.md && \
git commit -m "Update CHANGELOG for v0.24.0" -S

    • tag and push v0.24.0 and 0-24-0-stable:

      git tag -s v0.24.0 -m "Version v0.24.0" && \
git push origin v0.24.0 0-24-0-stable

    • checkout to master and merge 0-24-0-stable into master (only this one time, to update CHANGELOG.md and make the tag available for ./scripts/prepare-changelog-entries.rb in next stable release), push master:

      git checkout master && \
git pull && \
git merge --no-ff 0-24-0-stable
# check that the only changes are in CHANGELOG.md
git push

  • update Runner's chart version used by GitLab: gitlab!50377 (merged)

    • create branch

      rrhelper \
    --dry-run \
    create-branch \
    gitlab-org/gitlab \
    update-gitlab-runner-helm-chart-to-0-24-0

    • create Merge Request

      rrhelper \
    --dry-run \
    create-merge-request \
    --release-checklist-issue 27282 \
    --replace-link-in-release-checklist-issue link_to_MR__gce_rhcvu \
    --milestone 13.8 \
    --runner-version 13.7.0 \
    --helm-chart-version 0.24.0 \
    gitlab-org/gitlab \
    update-gitlab-runner-helm-chart-to-0-24-0 \
    master \
    runner-helm-chart-upgrade-in-gitlab \
    "Update GitLab Runner Helm Chart to 0.24.0/13.7.0"

    • Adjust and apply the patch to the GitLab CE sources

      See the patch draft

      • Adjust the following patch (set proper current version; set the Merge Request ID in the CHANGELOG entry file) and save it at /tmp/patch.gitlab.

        cat > /tmp/patch.gitlab << EOF
diff --git a/app/models/clusters/applications/runner.rb b/app/models/clusters/applications/runner.rb
index 0c0247da1fb..f17da0bb7b1 100644
--- a/app/models/clusters/applications/runner.rb
+++ b/app/models/clusters/applications/runner.rb
@@ -3,7 +3,7 @@
 module Clusters
   module Applications
     class Runner < ApplicationRecord
-      VERSION = '0.23.0'
+      VERSION = '0.24.0'

       self.table_name = 'clusters_applications_runners'

diff --git a/changelogs/unreleased/update-gitlab-runner-helm-chart-to-0-24-0.yml b/changelogs/unreleased/update-gitlab-runner-helm-chart-to-0-24-0.yml
new file mode 100644
index 00000000000..7d92929221f
--- /dev/null
+++ b/changelogs/unreleased/update-gitlab-runner-helm-chart-to-0-24-0.yml
@@ -0,0 +1,5 @@
+---
+title: Update GitLab Runner Helm Chart to 0.24.0
+merge_request: XXX
+author:
+type: other
EOF

      • Go to your local GitLab CE working directory:

        git checkout master && \
    git pull && \
    git checkout update-gitlab-runner-helm-chart-to-0-24-0 && \
    git apply /tmp/patch.gitlab && \
    git add . && \
    git commit -m "Update GitLab Runner Helm Chart to 0.24.0"

      • Push changes

        git push -u origin update-gitlab-runner-helm-chart-to-0-24-0

  • update Runner's chart version used by Cluster Applications: gitlab-org/cluster-integration/cluster-applications!133 (merged)

    • create branch

      rrhelper \
    --dry-run \
    create-branch \
    gitlab-org/cluster-integration/cluster-applications \
    update-gitlab-runner-helm-chart-to-0-24-0

    • create Merge Request

      rrhelper \
    --dry-run \
    create-merge-request \
    --release-checklist-issue 27282 \
    --replace-link-in-release-checklist-issue link_to_MR__ghc_rhcvu \
    --milestone 13.7 \
    --runner-version 13.7.0 \
    --helm-chart-version 0.24.0 \
    gitlab-org/cluster-integration/cluster-applications \
    update-gitlab-runner-helm-chart-to-0-24-0 \
    master \
    runner-helm-chart-upgrade-in-cluster-applications \
    "feat: Update GitLab Runner Helm Chart to 0.24.0/13.7.0"

    • Adjust and apply the patch to the GitLab Helm Chart sources

      See the patch draft

      • Adjust the following patch (set proper current version; set the Merge Request ID in the CHANGELOG entry file) and save it at /tmp/patch.gitlab-cluster-applications.

        cat > /tmp/patch.gitlab-cluster-applications << EOF
diff --git a/src/default-data/gitlab-runner/helmfile.yaml b/src/default-data/gitlab-runner/helmfile.yaml
index d839d7f..f358afa 100644
--- a/src/default-data/gitlab-runner/helmfile.yaml
+++ b/src/default-data/gitlab-runner/helmfile.yaml
@@ -13,7 +13,7 @@ releases:
 - name: runner
   namespace: {{ requiredEnv "TILLER_NAMESPACE" }}
   chart: gitlab/gitlab-runner
-  version: 0.23.0
+  version: 0.24.0
   installed: {{ .Values.gitlabRunner.installed }}
   missingFileHandler: Warn
   values:
EOF

      • Go to your local GitLab Cluster Application working directory:

        git checkout master && \
    git pull && \
    git checkout update-gitlab-runner-helm-chart-to-0-24-0 && \
    git apply /tmp/patch.gitlab-cluster-applications && \
    git add . && \
    git commit -m "feat: Update GitLab Runner Helm Chart to 0.24.0"

      • Push changes

        git push -u origin update-gitlab-runner-helm-chart-to-0-24-0

At 22nd - the official day of GitLab release

  • wait for Pipeline for v13.7.0 to pass pipeline status

    • add all required fixes to make v13.7.0 passing

  • deploy v13.7.0

    Detailed description of Runners deployment for GitLab.com CI fleet can be found in the runbook.

    • at release day: to prmX runners

      • go to your local chef-repo working directory and execute:

        knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-prm-runners-to-13-7-0

      • update version

        $EDITOR roles/gitlab-runner-prm.json

        In the role definition prepare the override_attributes entry. It should be placed at the top of the file:

        "override_attributes": {
  "cookbook-gitlab-runner": {
    "gitlab-runner": {
        "repository": "gitlab-runner",
        "version": "13.7.0"
    }
  }
},

      • git add roles/gitlab-runner-prm.json && git commit -m "Update prmX runners to v13.7.0"

      • git push -u origin update-prm-runners-to-13-7-0

      • after pushing the branch, create and manage to merge the chef-repo MR

      • check the production_dry_run job if it tries to update only the changed role

      • start the manual apply to prod job

      • after the job is finished execute:

        knife ssh -C 1 -afqdn 'roles:gitlab-runner-prm' -- 'sudo -i /root/runner_upgrade.sh' &
time wait

    • next day (if no problems): to rest of the runners

      • go to your local chef-repo working directory and execute:

        knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i /root/runner_upgrade.sh stop_chef'
knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm OR roles:org-ci-base-runner' -- 'sudo -i systemctl is-active chef-client'
git checkout master && git pull
git checkout -b update-runners-to-13-7-0

      • update version for gsrm/srm

        $EDITOR roles/gitlab-runner-base.json

        In the role definition prepare the gitlab-runner entry:

        "cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "gitlab-runner",
    "version": "13.7.0"
  }
}

      • update version for org-ci

        $EDITOR roles/org-ci-base-runner.json

        In the role definition prepare the gitlab-runner entry:

        "cookbook-gitlab-runner": {
  "gitlab-runner": {
    "repository": "gitlab-runner",
    "version": "13.7.0"
  }
}

      • remove overrides from prmX runners

        $EDITOR roles/gitlab-runner-prm.json

      • git add roles/gitlab-runner-prm.json roles/gitlab-runner-base.json roles/org-ci-base-runner.json && git commit -m "Update runners to v13.7.0"

      • git push -u origin update-runners-to-13-7-0

      • after pushing the branch, create and manage to merge the chef-repo MR

      • check the production_dry_run job if it tries to update only the changed role

      • start the manual apply to prod job

      • after the job is finished execute (we're not touching prmX - they are already updated):

        knife ssh -C1 -afqdn 'roles:gitlab-runner-gsrm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:gitlab-runner-srm' -- 'sudo -i /root/runner_upgrade.sh' &
knife ssh -C1 -afqdn 'roles:org-ci-base-runner' -- 'sudo -i /root/runner_upgrade.sh' &
time wait

Before next 7th

  • chose a release manager

  • create a Pick into 0.25 label in Helm Chart's project

    rrhelper \
    --dry-run \
    create-label \
    6329679 \
    "Pick into 0.25" \
    "Merge Requests to pick into 0-25-0 stable branch" \
    "#00C8CA"

  • create the new Release Checklist issue: #27420 (closed)

    rrhelper \
    create-release-checklist \
    --do-not-create \
    --non-interactive \
    --replace-link-in-previous-release-checklist-issue link_to_new_checklist_issue \
    --release-manager-handle release_manager \
    --previous-release-checklist-issue 27282
Edited by Tomasz Maczukin