Release 17.5

First steps

First Staging Rollback Practice

Date to be Completed: 2024-10-01

Second Staging Rollback Practice

Date to be Completed:

Hot patching production practice

2024-09-23- abandoned - pls refer to: #13193 (comment 2124365035)

2024-09-26 - succeeded!

The work here is to be done by the next release managers. Please tag them to the item below.

Ruby 3.2 Rollout - Tuesday, Oct 1

We have a change request to rollout Ruby 3.2 to production on Oct 1: gitlab-com/gl-infra/production#18519 (closed)

  • No PDM should be run on Oct 1 and 2 to allow sufficient window for potential rollback.
  • Deployments will be blocked around 15:00 - 21:00 UTC while the CR is in progress.

Check Pre environment

Date to be Completed: Friday, Oct 11

Up until Friday, Oct 11

  • Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
  • Push any successful deploy to canary into production after some time has passed (preferably 1h).
  • Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
    1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
    2. If permission is granted, set the OVERRIDE_PRODUCTION_CHECKS_REASON as a variable in the manual promote job. The value of the variable should be the reason why production checks are being overridden. See gitlab-com-deployer.md#skipping-production-promotion-checks for more info.
    3. This will post a comment into this issue and begin the deployment
    4. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

Release Preparation

Initial preparation day: Friday, Oct 11

  • Find the latest sha that made it into production successfully: 9ae09dab597df501719bace17a7210f78161eea4

  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

  • Manually create the pipeline to update monthly release status to announced.

  • Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 17th:

    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 17th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 17.5`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`
  * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1"

  • Verify that the release information dashboard reflects the accurate status "announced" for the active release version (17.5).

    • The updated status can take up to 15 minutes to display on the dashboard once the pipeline to create the metric is finished.

Candidate announcement day: Monday, Oct 14

  • Log latest auto-deploy branch: 17-5-auto-deploy-2024101408

  • Ensure this build makes it through into production

  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

  • Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release:

    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 17th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 17.5`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`
  * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1"

RC tag day: Tuesday, Oct 15

  • Determine what is the last auto deploy branch to have deployed to production and add it here: 17.5.202410142206

  • If you plan to use the latest commit deployed to production for the various components to create the RC, make sure:

  • Create a RC version to ensure that the final version builds correctly

    # In Slack:
/chatops run release tag 17.5.0-rc42

This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:

/chatops run release tag 17.5.0-rc42 --gitlab-sha=XXX

This will then use XXX as the SHA to create the GitLab stable branches.

NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

  • Verify that the CE stable branch contains the right commits

    • There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
    • The sync commit will have the message "Add latest changes from gitlab-org/gitlab@17-5-stable-ee"

  • Verify that the pipelines are green

    # In Slack:
/chatops run release status 17.5.0-rc42

  • Verify that the RC has been deployed to the pre environment

    • Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
    • If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.

  • Verify that the release information dashboard reflects the accurate status "RC Tagged" for the active release version (17.5).

    • The updated status can take up to 15 minutes to display on the dashboard once the RC is tagged.

  • Notify Engineering Managers and developers that final candidate has been created:

    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 17th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-5-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 17.5`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`
  * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1"

Tag day: Wednesday, Oct 16

  • Start the monthly_release_tag_day:prepare job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3789653
    • Jobs to verify the integrity of the stable branches (monthly_release_tag_day:ensure_stable_branches_green) and the mirror status (monthly_release_tag_day:mirror_status) will start automatically.
  • Once the above jobs are green, trigger the monthly_release_tag_day:start_tag job: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3789653
    • This job will tag the final 17.5.0 version.
  • The monthly_release_tag_day:check_omnibus_packages_tagging job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3789653 will be triggered 90 minutes after tagging completes. However, if possible, you can start it earlier. In case the job fails, troubleshoot the issue, and rerun the job.
  • Check that the CNG Images are built: CNG builds.
  • Check that the Helm charts for 17.5.0 are built. Note that the Helm chart version is 9 major versions behind GitLab version. For example, GitLab 17.3 has the corresponding Helm chart 8.3, so the stable branch is 8-3-stable.
  • After tagging, the release.gitlab.net instance was automatically updated with the latest monthly release. Corroborate if the deployment was successful by verifying that the monthly_release_tag_day:verify_release_deploy job succeeded.
    • A Slack notification with Validate release.gitlab.net running 17.5.0 has successfully executed will be sent to #f_upcoming_release.
  • [-] Validate 17.5.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green. See #13193 (comment 2161654399)

Past this point, no new code can be added to the release that was not included in the final RC.

Release day: Thursday, Oct 17

Final release is tagged, so any changes will have to initiate a patch release. Reminder: We have a soft PCL today, coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.

  • At 13:00 UTC, start the monthly_release_release_day:start job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3789653. An announcement will be sent and the publish will happen 10 minutes later automatically. In case you want to publish the release immediately, skip the delay time of the job monthly_release_release_day:publish.

    • ⚠ Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
    • If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too

  • At 14:10 UTC:

    • Start the monthly_release_verify:start job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3801175
    • Make sure the monthly_release:verify stage has completed successfully.
    • Check that the CNG Images are published: CNG builds. Each version has 4 tags for ee, fips, ubi and the branch itself.
    • Check that the Helm charts for 17.5.0 are published. Note that the Helm chart version is 9 major versions behind GitLab version. For example, GitLab 17.3 has the corresponding Helm chart 8.3, so the stable branch is 8-3-stable.
    • Post an update about the status in #f_upcoming_release 
    :mega: 17.5.0 is published and publicly available

Release Certification

The release certification process may apply to this release. cc @gitlab-com/gl-security/product-security/federal-application-security

Edited by Mayra Cabrera