Release 14.8

First steps

  • Change #f_upcoming_release topic with /topic 14.8.0: <link_to_this_issue>
  • Adjust the auto-deploy branch schedule based on the Release Manager's working hours. gitlab-com/www-gitlab-com!98011 (merged)
  • [-] Consider planned PCLs and modify the dates below to accurately reflect the plan of action. => No plan PCL so far
  • [-] If there is a Family & Friends day this month, add notes for pausing deployments before the day starts, and unpausing them before the next business day. See the documentation. => Next family and friends is on the 25th
  • Update this issue with three planned dates for recurring Staging rollback practice. Consider spreading these across timezones to share the knowledge.
    • Set Due Date for this Issue to the first practice session
  • [-] Check for any deprecations and see if we are possibly affected in our k8s or chef config. with removal: '14.8' that might affect our configuration - otherwise the next auto-deploy after a monthly release might fail with a deprecation failure

First Staging Rollback Practice

Feb 7th

Second Staging Rollback Practice:

Feb 9th

Third Staging Rollback Practice

Feb 11th

Up until the 16th

  • Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
  • Push any successful deploy to canary into production after some time has passed (preferably 1h).
  • Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
    1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
    2. If permission is granted, utilize the following command to initiate an overridden promotion:
    /chatops run deploy <VERSION> --production --ignore-production-checks 'deployment approved by on call SRE'
    1. This will post a comment into this issue and begin the deployment
    2. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

16th

If this date is on a weekend, do this work on the next working day

  • Find the latest sha that made it into production successfully: fc1fbf2941bffca255d92911ebc84bf03c23f31f
  • Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 22nd: 
    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/commits/fc1fbf2941bffca255d92911ebc84bf03c23f31f.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 14.8`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

17th

If this date is on a weekend, do this work on the last Friday before the 18th.

  • Log latest auto-deploy branch: https://gitlab.com/gitlab-org/security/gitlab/-/commits/14-8-auto-deploy-2022021715
  • Ensure this build makes it through into production
  • Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release: 
    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/a3a124db89bdbed8c69b9ad0cd3781c5bc514098
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 14.8`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

18th: two working days before the release

  • Create an RC version to ensure that the final version builds correctly

    # In Slack:
/chatops run release tag 14.8.0-rc42 --gitlab-sha=a3a124db89bdbed8c69b9ad0cd3781c5bc514098

    This will use a3a124db89bdbed8c69b9ad0cd3781c5bc514098 (candidate commit) as the SHA to create the GitLab stable branches.

    NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

    Alternatively, use the latest commit deployed to production to build the RC:

    # In Slack:
/chatops run release tag 14.8.0-rc42

  • Verify that the CE stable branch contains the right commits

    • There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
    • The sync commit will have the message "Add latest changes from gitlab-org/gitlab@14-8-stable-ee"

  • Notify Engineering Managers and developers that final candidate has been created:

    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/14-8-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 14.8`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

  • Verify that the RC has been deployed to the pre environment

    • Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
    • If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.

21st: one day before the release

Instructions for manual deploy 
    ```sh
    # In Slack:
    /chatops run deploy 14.8.0-ee.0 --release
    ```
  • Validate 14.8.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

22nd: release day

Final release is tagged, so any changes will have to initiate a patch release.

  • At 13:00 UTC, post an update about the package building status in #f_upcoming_release 
    :mega: Packages for 14.8.0 are built and will be published at 13:30UTC
  • At 13:30 UTC:
    • ⚠ Make sure that neither packages nor the blog post get published earlier than 13:30UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
    • Publish the packages via ChatOps: 
      # In Slack:
/chatops run publish 14.8.0
    • If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
  • At 14:10 UTC:
    • Verify that EE packages appear on packages.gitlab.com: EE (should contain 14 packages)
    • Verify that CE packages appear on packages.gitlab.com: CE (should contain 13 packages)
    • Verify that Docker images appear on hub.docker.com: EE / CE
    • Post an update about the status in #f_upcoming_release 
    :mega: 14.8.0 is published and publicly available
    • Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC
    • Create the 14.8.0 version on version.gitlab.com
Edited by Ahmad Tolba