GitLab Runner 12.2 release checklist

Previous release checklist issue: #4448 (closed)

GitLab Runner Release manager: @tmaczukin

Release blog post MR: gitlab-com/www-gitlab-com!27164 (merged)

Runner entries need to be added to blog post until: 2019-08-13

First working day after 7th - v12.2.0-rc1 release

• check if Pipeline for master is passing:

  • add all required fixes to make master Pipeline passing

• git checkout master && git pull in your local working copy!

• prepare CHANGELOG entries

  ./scripts/prepare-changelog-entries.rb

  Copy the lines to the beginning of CHANGELOG.md file and add a proper header:

  v12.2.0-rc1 (TODAY_DATE_HERE)

• add v12.2.0-rc1 CHANGELOG entries and commit

  git add CHANGELOG.md && \
  git commit -m "Update CHANGELOG for v12.2.0-rc1" -S

• tag and push v12.2.0-rc1:

  git tag -s v12.2.0-rc1 -m "Version v12.2.0-rc1" && \
  git push origin v12.2.0-rc1

• create and push 12-2-stable branch:

  git checkout -b 12-2-stable && \
  git push -u origin 12-2-stable

• checkout to master, update VERSION file to 12.3.0 and push master:

  git checkout master; echo -n "12.3.0" > VERSION && \
  git add VERSION && \
  git commit -m "Bump version to 12.3.0" -S && \
  git push

• wait for Pipeline for v12.2.0-rc1 to pass

  • add all required fixes to make v12.2.0-rc1 passing

• update runner helm chart to use v12.2.0-rc1 version

  • check if Pipeline for master is passing:

    • add all required fixes to make master Pipeline passing

  • go to your local working copy of https://gitlab.com/charts/gitlab-runner

  • git checkout master && git pull in your local working copy!

  • set Helm Chart to use v12.2.0-rc1 version of Runner

    • create new branch, update Runner version and push the branch:

      git checkout -b update-runner-to-12-2-0-rc1 && \
      sed -i".bak" "s/^appVersion: .*/appVersion: 12.2.0-rc1/" Chart.yaml && \
      rm Chart.yaml.bak && \
      git add Chart.yaml && \
      git commit -m "Bump used Runner version to 12.2.0-rc1" -S && \
      git push -u origin update-runner-to-12-2-0-rc1

    • create Merge Request pointing master: link_to_MR__rhc_urv_rc1

      rrhelper \
          --dry-run \
          create-merge-request \
          --release-checklist-issue 4576 \
          --replace-link-in-release-checklist-issue link_to_MR__rhc_urv_rc1 \
          --milestone 12.2 \
          --runner-version 12.2.0-rc1 \
          charts/gitlab-runner \
          update-runner-to-12-2-0-rc1 \
          master \
          runner-version-bump-in-runner-helm-chart \
          "Bump used Runner version to 12.2.0-rc1"

    • manage to merge the MR

  • check if Pipeline for master is passing:

    • add all required fixes to make master Pipeline passing

  • git checkout master && git pull in your local working copy!

  • prepare CHANGELOG entries

    ./scripts/prepare-changelog-entries.rb

    Copy the lines to the beginning of CHANGELOG.md file and add a proper header:

    ## v0.8.0-rc1 (TODAY_DATE_HERE)

  • add v0.8.0-rc1 CHANGELOG entries and commit

    git add CHANGELOG.md && \
    git commit -m "Update CHANGELOG for v0.8.0-rc1" -S

  • bump version of the Helm Chart to 0.8.0-rc1

    sed -i".bak" "s/^version: .*/version: 0.8.0-rc1/" Chart.yaml && \
    rm Chart.yaml.bak && \
    git add Chart.yaml && \
    git commit -m "Bump version to 0.8.0-rc1" -S

  • tag and push v0.8.0-rc1:

    git tag -s v0.8.0-rc1 -m "Version v0.8.0-rc1" && \
    git push origin v0.8.0-rc1

  • create and push 0-8-0-stable branch:

    git checkout -b 0-8-0-stable && \
    git push -u origin 0-8-0-stable

  • checkout to master, bump version of the Helm Chart to 0.9.0-beta and push master:

    git checkout master && \
    sed -i".bak" "s/^version: .*/version: 0.9.0-beta/" Chart.yaml && \
    rm Chart.yaml.bak && \
    git add Chart.yaml && \
    git commit -m "Bump version to 0.9.0-beta" -S && \
    git push

• deploy v12.2.0-rc1

  Detailed description of Runners deployment for GitLab.com CI fleet can be found in the runbook.

  • at RC1 release day: to prmX runners

    • go to your local chef-repo working directory and execute:

      knife ssh -afqdn 'roles:gitlab-runner-prm' -- sudo systemctl stop chef-client
      knife ssh -afqdn 'roles:gitlab-runner-prm' -- sudo systemctl is-active chef-client
      git checkout master && git pull
      git checkout -b update-prm-runners-to-12-2-0-rc1

    • update version

      $EDITOR roles/gitlab-runner-prm.json

      In the role definition prepare the override_attributes entry. It should be placed at the top of the file:

      "override_attributes": {
        "cookbook-gitlab-runner": {
          "gitlab-runner": {
              "repository": "unstable",
              "version": "12.2.0-rc1"
          }
        }
      },

    • git add roles/gitlab-runner-prm.json && git commit -m "Update prmX runners to v12.2.0-rc1"

    • git push -u origin update-prm-runners-to-12-2-0-rc1

    • after pushing the branch, create and manage to merge the chef-repo MR

    • check the apply to staging job if it tries to update only the changed role

    • start the manual apply to prod job

    • after the job is finished execute:

      knife ssh -C 1 -afqdn 'roles:gitlab-runner-prm' -- sudo /root/runner_upgrade.sh &
      time wait

  • next day (if no problems): to rest of the runners

    • go to your local chef-repo working directory and execute:

      knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm' -- sudo systemctl stop chef-client
      knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm' -- sudo systemctl is-active chef-client
      git checkout master && git pull
      git checkout -b update-runners-to-12-2-0-rc1

    • update version

      $EDITOR roles/gitlab-runner-base.json

      In the role definition prepare the gitlab-runner entry:

      "cookbook-gitlab-runner": {
        "gitlab-runner": {
          "repository": "unstable",
          "version": "12.2.0-rc1"
        }
      }

    • remove overrides from prmX runners

      $EDITOR roles/gitlab-runner-prm.json

    • git add roles/gitlab-runner-prm.json roles/gitlab-runner-base.json && git commit -m "Update runners to v12.2.0-rc1"

    • git push -u origin update-runners-to-12-2-0-rc1

    • after pushing the branch, create and manage to merge the chef-repo MR

    • check the apply to staging job if it tries to update only the changed role

    • start the manual apply to prod job

    • after the job is finished execute (we're not touching prmX - they are already updated):

      knife ssh -C1 -afqdn 'roles:gitlab-runner-builder' -- sudo /root/runner_upgrade.sh &
      knife ssh -C1 -afqdn 'roles:gitlab-runner-gsrm' -- sudo /root/runner_upgrade.sh &
      knife ssh -C1 -afqdn 'roles:gitlab-runner-srm' -- sudo /root/runner_upgrade.sh &
      time wait

New features window is closed - things not merged into master up to this day, will be released with next release.

7 working days before 22th (2019-08-13)

• prepare GitLab Runner entries for the release blog post. Items can be generated with ./scripts/changelog2releasepost | less (executed in Runner's local working copy directory)

• add release entry:

  Add description to the SECONDARY FEATURES list using following template:

  - name: GitLab Runner 12.2
    available_in: [core, starter, premium, ultimate]
    documentation_link: 'https://docs.gitlab.com/runner'
    documentation_text: "Read through the documentation of GitLab Runner"
    description: |
      We're also releasing GitLab Runner 12.2 today! GitLab Runner is the open source project
      that is used to run your CI/CD jobs and send the results back to GitLab.
  
      ##### Most interesting changes:
  
      * [Configuration file template for registration command](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1263)
      * [Prevent running multiple instances of the gitlab-runner process using the same configuration file](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1496)
  
      List of all changes can be found in GitLab Runner's [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-runner/blob/v12.2.0/CHANGELOG.md).

Pick into stable branch for RC2 phase

At this moment, until the next RC creation, we're in the pick into stable phase. The release manager should periodically review the list of merged MRs with a pick into X.Y label:

• GitLab Runner ~"Pick into 12.2" Merge Requests
• GitLab Runner Helm Chart ~"Pick into 0.8" Merge Requests

GitLab Runner MRs:

GitLab Runner's Helm Chart MRs:

For each such Merge Request the release manager should (reveal for details)

1. Find the merge commit for the Merge Request

2. Copy the commit reference

3. Checkout to the branch:

   • git checkout 12-2-stable && git pull for GitLab Runner, or
   • git checkout 0-8-0-stable && git pull for GitLab Runner's Helm Chart

4. Pick the merge commit with:

   git cherry-pick -m 1 [merge commit SHA here]

5. Remove the Pick into X.Y label from the Merge Request and leave a note that it was picked:

• for GitLab Runner use this message:

  /unlabel ~"Pick into 12.2"
  
  The MR was picked into `12-2-stable` branch and will be released with `v12.2.0-rc2`

• for GitLab Runner's Helm Chart use this message:

  /unlabel ~"Pick into 0.8"
  
  The MR was picked into `0-8-0-stable` branch and will be released with `v0.8.0-rc2`

To prepare the `Pick into ...` MRs list execute:

rrhelper \
    list-pick-mrs-for-rc \
    --non-interactive \
    --do-not-create \
    --major 12 \
    --minor 2  \
    --helm-chart-major 0 \
    --helm-chart-minor 8 \
    4576 2

RC2 (if needed)

If any commit was picked into the stable branch since RC1 was released, RC2 version must be released no later than at 18th day of the month or immediately if changes were picked after 18th.

If you need to create RC2 version, then please execute this and follow the created checklist for RC2:

rrhelper \
    create-next-rc-checklist \
    --non-interactive \
    --do-not-create \
    --major 12 \
    --minor 2  \
    --helm-chart-major 0 \
    --helm-chart-minor 8 \
    4576 2

At 20th - the release day of GitLab Runner

• check if Pipeline for 12-2-stable is passing:

  • add all required fixes to make 12-2-stable Pipeline passing

• git checkout 12-2-stable && git pull in your local working copy!

• merge all RCx CHANGELOG entries into release entry

  Put a proper header at the begining:

  v12.2.0 (TODAY_DATE_HERE)

• add v12.2.0 CHANGELOG entries and commit

  git add CHANGELOG.md && \
  git commit -m "Update CHANGELOG for v12.2.0" -S

• tag and push v12.2.0 and 12-2-stable:

  git tag -s v12.2.0 -m "Version v12.2.0" && \
  git push origin v12.2.0 12-2-stable

• checkout to master and merge 12-2-stable into master (only this one time, to update CHANGELOG.md and make the tag available for ./scripts/prepare-changelog-entries.rb in next stable release), push master:

  git checkout master && \
  git pull && \
  git merge --no-ff 12-2-stable
  # check that the only changes are in CHANGELOG.md
  git push

• update runner helm chart to use v12.2.0 version

  • check if Pipeline for 0-8-0-stable is passing:

    • add all required fixes to make 0-8-0-stable Pipeline passing

  • go to your local working copy of https://gitlab.com/charts/gitlab-runner

  • git checkout 0-8-0-stable && git pull in your local working copy!

  • set Helm Chart to use v12.2.0 version of Runner

    sed -i".bak" "s/^appVersion: .*/appVersion: 12.2.0/" Chart.yaml && \
    rm Chart.yaml.bak && \
    git add Chart.yaml && \
    git commit -m "Bump used Runner version to 12.2.0" -S

  • merge all RCx CHANGELOG entries into release entry

    Put a proper header at the begining:

    ## v0.8.0 (TODAY_DATE_HERE)

  • add v0.8.0 CHANGELOG entries and commit

    git add CHANGELOG.md && \
    git commit -m "Update CHANGELOG for v0.8.0" -S

  • bump version of the Helm Chart to 0.8.0

    sed -i".bak" "s/^version: .*/version: 0.8.0/" Chart.yaml && \
    rm Chart.yaml.bak && \
    git add Chart.yaml && \
    git commit -m "Bump version to 0.8.0" -S

  • tag and push v0.8.0 and 0-8-0-stable:

    git tag -s v0.8.0 -m "Version v0.8.0" && \
    git push origin v0.8.0 0-8-0-stable

  • checkout to master and merge 0-8-0-stable into master (only this one time, to update CHANGELOG.md and make the tag available for ./scripts/prepare-changelog-entries.rb in next stable release), push master:

    git checkout master && \
    git pull && \
    git merge --no-ff --no-commit 0-8-0-stable && \
    git reset . && \
    git add CHANGELOG.md && \
    git checkout -- .
    # check that the only changes are in CHANGELOG.md
    git commit -S && \
    git push

• update Runner's chart version used by GitLab: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32289

  • create branch

    rrhelper \
        --dry-run \
        create-branch \
        gitlab-org/gitlab-ce \
        update-gitlab-runner-helm-chart-to-0-8-0

  • create Merge Request

    rrhelper \
        --dry-run \
        create-merge-request \
        --release-checklist-issue 4576 \
        --replace-link-in-release-checklist-issue link_to_MR__gce_rhcvu \
        --milestone 12.3 \
        --runner-version 12.2.0 \
        --helm-chart-version 0.8.0 \
        gitlab-org/gitlab-ce \
        update-gitlab-runner-helm-chart-to-0-8-0 \
        master \
        runner-helm-chart-upgrade-in-gitlab-ce \
        "Update GitLab Runner Helm Chart to 0.8.0/12.2.0"

  • Adjust and apply the patch to the GitLab CE sources

    See the patch draft

    • Adjust the following patch (set proper current version; set the Merge Request ID in the CHANGELOG entry file) and save it at /tmp/patch.gitlab-ce.

      cat > /tmp/patch.gitlab-ce << EOF
      diff --git a/app/models/clusters/applications/runner.rb b/app/models/clusters/applications/runner.rb
      index 0c0247da1fb..f17da0bb7b1 100644
      --- a/app/models/clusters/applications/runner.rb
      +++ b/app/models/clusters/applications/runner.rb
      @@ -3,7 +3,7 @@
       module Clusters
         module Applications
           class Runner < ApplicationRecord
      -      VERSION = '0.7.0'.freeze
      +      VERSION = '0.8.0'.freeze
      
             self.table_name = 'clusters_applications_runners'
      
      diff --git a/changelogs/unreleased/update-gitlab-runner-helm-chart-to-0-8-0.yml b/changelogs/unreleased/update-gitlab-runner-helm-chart-to-0-8-0.yml
      new file mode 100644
      index 00000000000..7d92929221f
      --- /dev/null
      +++ b/changelogs/unreleased/update-gitlab-runner-helm-chart-to-0-8-0.yml
      @@ -0,0 +1,5 @@
      +---
      +title: Update GitLab Runner Helm Chart to 0.8.0
      +merge_request: XXX
      +author:
      +type: other
      EOF

    • Go to your local GitLab CE working directory:

      git checkout master && \
          git pull && \
          git checkout update-gitlab-runner-helm-chart-to-0-8-0 && \
          git apply /tmp/patch.gitlab-ce && \
          git add . && \
          git commit -m "Update GitLab Runner Helm Chart to 0.8.0"

    • Push changes

      git push -u origin update-gitlab-runner-helm-chart-to-0-8-0

• update Runner's chart version used by GitLab chart: https://gitlab.com/charts/gitlab/merge_requests/912

  • create branch

    rrhelper \
        --dry-run \
        create-branch \
        charts/gitlab \
        update-gitlab-runner-helm-chart-to-0-8-0

  • create Merge Request

    rrhelper \
        --dry-run \
        create-merge-request \
        --release-checklist-issue 4576 \
        --replace-link-in-release-checklist-issue link_to_MR__ghc_rhcvu \
        --milestone 12.2 \
        --runner-version 12.2.0 \
        --helm-chart-version 0.8.0 \
        charts/gitlab \
        update-gitlab-runner-helm-chart-to-0-8-0 \
        master \
        runner-helm-chart-upgrade-in-gitlab-helm-chart \
        "Update GitLab Runner Helm Chart to 0.8.0/12.2.0"

  • Adjust and apply the patch to the GitLab Helm Chart sources

    See the patch draft

    • Adjust the following patch (set proper current version; set the Merge Request ID in the CHANGELOG entry file) and save it at /tmp/patch.gitlab-helm-chart.

      cat > /tmp/patch.gitlab-helm-chart << EOF
      diff --git a/changelogs/unreleased/update-gitlab-runner-to-0-8-0.yml b/changelogs/unreleased/update-gitlab-runner-to-0-8-0.yml
      new file mode 100644
      index 00000000..5fd73668
      --- /dev/null
      +++ b/changelogs/unreleased/update-gitlab-runner-to-0-8-0.yml
      @@ -0,0 +1,5 @@
      +---
      +title: Update gitlab-runner to 0.8.0/12.2.0
      +merge_request: XXX
      +author:
      +type: other
      diff --git a/requirements.yaml b/requirements.yaml
      index 82a0192a..c6293a8d 100644
      --- a/requirements.yaml
      +++ b/requirements.yaml
      @@ -13,6 +13,6 @@ dependencies:
         repository: https://kubernetes-charts.storage.googleapis.com/
         condition: postgresql.install
       - name: gitlab-runner
      -  version: 0.7.0
      +  version: 0.8.0
         repository: https://charts.gitlab.io/
         condition: gitlab-runner.install
      EOF

    • Go to your local GitLab Helm Chart working directory:

      git checkout master && \
          git pull && \
          git checkout update-gitlab-runner-helm-chart-to-0-8-0 && \
          git apply /tmp/patch.gitlab-helm-chart && \
          git add . && \
          git commit -m "Update GitLab Runner Helm Chart to 0.8.0"

    • Push changes

      git push -u origin update-gitlab-runner-helm-chart-to-0-8-0

At 22nd - the official day of GitLab release

• wait for Pipeline for v12.2.0 to pass

  • add all required fixes to make v12.2.0 passing

• deploy v12.2.0

  Detailed description of Runners deployment for GitLab.com CI fleet can be found in the runbook.

  • at release day: to prmX runners

    • go to your local chef-repo working directory and execute:

      knife ssh -afqdn 'roles:gitlab-runner-prm' -- sudo systemctl stop chef-client
      knife ssh -afqdn 'roles:gitlab-runner-prm' -- sudo systemctl is-active chef-client
      git checkout master && git pull
      git checkout -b update-prm-runners-to-12-2-0

    • update version

      $EDITOR roles/gitlab-runner-prm.json

      In the role definition prepare the override_attributes entry. It should be placed at the top of the file:

      "override_attributes": {
        "cookbook-gitlab-runner": {
          "gitlab-runner": {
              "repository": "gitlab-runner",
              "version": "12.2.0"
          }
        }
      },

    • git add roles/gitlab-runner-prm.json && git commit -m "Update prmX runners to v12.2.0"

    • git push -u origin update-prm-runners-to-12-2-0

    • after pushing the branch, create and manage to merge the chef-repo MR

    • check the apply to staging job if it tries to update only the changed role

    • start the manual apply to prod job

    • after the job is finished execute:

      knife ssh -C 1 -afqdn 'roles:gitlab-runner-prm' -- sudo /root/runner_upgrade.sh &
      time wait

  • next day (if no problems): to rest of the runners

    • go to your local chef-repo working directory and execute:

      knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm' -- sudo systemctl stop chef-client
      knife ssh -afqdn 'roles:gitlab-runner-gsrm OR roles:gitlab-runner-srm' -- sudo systemctl is-active chef-client
      git checkout master && git pull
      git checkout -b update-runners-to-12-2-0

    • update version

      $EDITOR roles/gitlab-runner-base.json

      In the role definition prepare the gitlab-runner entry:

      "cookbook-gitlab-runner": {
        "gitlab-runner": {
          "repository": "gitlab-runner",
          "version": "12.2.0"
        }
      }

    • remove overrides from prmX runners

      $EDITOR roles/gitlab-runner-prm.json

    • git add roles/gitlab-runner-prm.json roles/gitlab-runner-base.json && git commit -m "Update runners to v12.2.0"

    • git push -u origin update-runners-to-12-2-0

    • after pushing the branch, create and manage to merge the chef-repo MR

    • check the apply to staging job if it tries to update only the changed role

    • start the manual apply to prod job

    • after the job is finished execute (we're not touching prmX - they are already updated):

      knife ssh -C1 -afqdn 'roles:gitlab-runner-builder' -- sudo /root/runner_upgrade.sh &
      knife ssh -C1 -afqdn 'roles:gitlab-runner-gsrm' -- sudo /root/runner_upgrade.sh &
      knife ssh -C1 -afqdn 'roles:gitlab-runner-srm' -- sudo /root/runner_upgrade.sh &
      time wait

Before next 7th

• chose a release manager

• create a Pick into 0.9 label in Helm Chart's project

• @release manager: create the new Release Checklist issue: [link to the ISSUE]

  rrhelper \
      create-release-checklist \
      --do-not-create \
      --non-interactive \
      --previous-release-checklist-issue 4576

  Ensure, that the above command handles:

  • adding link to the release blog post's MR

  • setting deadline for add entries to release blog post

    Please check what deadline is set for General Contributions section in the release blog post Merge Request. It should be 6th working day before the 22nd. In that case we can set our deadline for 7th working day before 22nd, however if the deadline from the MR is earlier, then use the earliest one.

  • updating the .Major and .Minor to a specific release version