Release 15.10

First steps

  • Change #f_upcoming_release topic with /topic 15.10.0: <link_to_this_issue>
  • Check whether the auto-deploy branch schedule fits the Release Managers' working hours. Adjust if needed and make sure any changes are updated in the handbook.
  • Consider planned PCLs and modify the dates below to accurately reflect the plan of action.
  • If there is a Family & Friends day this month, add notes for pausing deployments before the day starts, and unpausing them before the next business day. See the documentation.
  • Update this issue with three planned dates for recurring Staging rollback practice. Consider spreading these across timezones to share the knowledge.
    • Set Due Date for this Issue to the first practice session
  • Check for any deprecations and see if we are possibly affected in our k8s or chef config. with removal: '15.10' that might affect our configuration - otherwise the next auto-deploy after a monthly release might fail with a deprecation failure
  • Confirm the due date set in the upcoming security release is achievable. Work with AppSec if the date needs to change

First Staging Rollback Practice

2022-03-03 #5119 (comment 1300550216)

Second Staging Rollback Practice

Date to be Completed: 2022-03-09

Third Staging Rollback Practice

Date to be Completed: 2022-03-15

Friday, March 3

Monday, March 6, 03:30 UTC

Tuesday, March 7, 9:00 UTC

Wednesday, March 8

  • Resume auto-deploys: /chatops run auto_deploy unpause
  • Note the PDM should not be executed on this day.

Thursday, March 9

Friday, March 10

  • (After 48 hours post ruby 3 upgrade: 2022-03-10 01:26 UTC) Resume execution of the PDM in the beginning of APAC timezone to allow Ruby 3 changes to sit in gprd for at least 48 hours

Up until the 15th

  • Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
  • Push any successful deploy to canary into production after some time has passed (preferably 1h).
  • Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
    1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
    2. If permission is granted, utilize the following command to initiate an overridden promotion:
    /chatops run deploy <VERSION> gprd --ignore-production-checks 'deployment approved by on call SRE'
    1. This will post a comment into this issue and begin the deployment
    2. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

16th

If this date is on a weekend, do this work on the next working day

  • Find the latest sha that made it into production successfully: 646786eee7f
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 22nd: 
    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/commits/<SHA>.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 15.10`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

17th

If this date is on a weekend, do this work on the last Friday before the 18th.

  • Log latest auto-deploy branch: 15-10-auto-deploy-2023031715
  • Ensure this build makes it through into production
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release: 
    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/commits/1a2b0f94f56
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 15.10`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

20th: two working days before the release

If this date is on a Sunday, do this work on the last Friday before the 20th. If it falls on a Friday or Saturday, move it to Thursday.

  • Determine what is the last auto deploy branch to have deployed to production and add it here:15-10-auto-deploy-2023032006

  • If you plan to use the latest commit deployed to production to create the RC, make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

  • Create a RC version to ensure that the final version builds correctly

    # In Slack:
/chatops run release tag 15.10.0-rc42

This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:

/chatops run release tag 15.10.0-rc42 --gitlab-sha=XXX

This will then use XXX as the SHA to create the GitLab stable branches.

NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

  • Verify that the CE stable branch contains the right commits

    • There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
    • The sync commit will have the message "Add latest changes from gitlab-org/gitlab@15-10-stable-ee"

  • Verify that the pipelines are green

    # In Slack:
/chatops run release status 15.10.0-rc42

  • Notify Engineering Managers and developers that final candidate has been created:

    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/15-10-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 15.10`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

  • Verify that the RC has been deployed to the pre environment

    • Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
    • If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.

21st: one day before the release

If this date is on a weekend, do this work on the Friday before that weekend.

Instructions for manual deploy 
    ```sh
    # In Slack:
    /chatops run deploy 15.10.0-ee.0 release
    ```
  • Validate 15.10.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

22nd: release day

Final release is tagged, so any changes will have to initiate a patch release. Reminder: We have a soft PCL today, coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.

  • At 13:00 UTC, post an update about the package building status in #f_upcoming_release 
    :mega: Packages for 15.10.0 are built and will be published at 13:10UTC
  • At 13:10 UTC:
    • ⚠ Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
    • Publish the packages via ChatOps: 
      # In Slack:
/chatops run publish 15.10.0
    • If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
  • At 14:10 UTC:
    • Verify the check-packages job completes successfully on the EE Pipeline
    • Verify the check-packages job completes successfully on the CE Pipeline
    • Verify that Docker images appear on hub.docker.com: EE / CE
    • Post an update about the status in #f_upcoming_release 
    :mega: 15.10.0 is published and publicly available
    • Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC
    • Create the 15.10.0 version on version.gitlab.com

Release Certification

The release certification process may apply to this release. cc @gitlab-com/gl-security/federal-application-security

Edited by Ahmad Tolba