Release 17.3

First steps

Change #f_upcoming_release topic with /topic 17.3.0: <link_to_this_issue>
Check whether the auto-deploy branch schedule fits the Release Managers' working hours. Adjust if needed and make sure any changes are updated in the handbook.
Check for any holiday, Family & Friends day or planned PCLs scheduled for this release:
- Adjust the release preparation days if necessary
- If there are Family and Friends day add notes for pausing deployments before the day starts and unpausing them before the next business day, see the auto-deploy documentation for details.
Update this issue with two planned dates for recurring Staging rollback practice and a planned date for Hot Patching Production practice. Consider spreading these across timezones to share the knowledge.
- Set Due Date for this Issue to the first practice session
Check for any deprecations: None

First Staging Rollback Practice

Date to be Completed: Friday 2024-07-26

Perform Staging Rollback Practice
Set Due date for this issue to the date to July 30th for the next step.

Soft PCL 2024-07-30

To support CR: gitlab-com/gl-infra/production#18299 (closed), we will block deployments and PDM operations for 24 hours.

2024-07-29 End of Day AMER - /chatops run deploy lock gprd-cny
2024-07-31 Start of Day EMEA - /chatops run deploy unlock gprd-cny
Set due date of this issue to 2024-08-01, for the next soft PCL.

Soft PCL 2024-08-01

To support CR: gitlab-com/gl-infra/production#18331 (closed), we will block deployments and PDM operations for 7 hours, between 0 - 7 UTC 2024-08-01.

2024-07-31 End of Day AMER
- /chatops run deploy lock gprd-cny
- /chatops run deploy lock gstg-cny
2024-08-01 after 7 UTC
- Coordinate with the coordinators of the CR to confirm the CR is done and we can continue auto-deploy
- /chatops run deploy unlock gstg-cny
- /chatops run deploy unlock gprd-cny
Set due date of this issue to August 2nd, for the next staging rollback practice session.

Hot patching production practice

NOTE: This practice is blocked by Validate token and fix notify-mr job for patche... (gitlab-com/gl-infra/delivery#20318 - closed). Please fix it first.

Date to be Completed: Friday 2024-08-02

The work here is to be done by the next release managers. Please tag them to the item below.

@madelacruz
Perform hot patching Practice
Hot Patching practice is documented: https://gitlab.com/gitlab-org/release/tasks/-/issues/11796#note_2020417959
Set Due date for this issue to the date of the next practice session. If none, set it to the release date.

Second Staging Rollback Practice

Date to be Completed: Friday 2024-08-09

Perform Staging Rollback Practice https://gitlab.com/gitlab-org/release/tasks/-/issues/11796#note_2036535451
Set Due date for this issue to the date of the next practice session. If none, set it to the release date.

Check Pre environment

Date to be Completed: Friday, Aug 9

The pre environment should be checked automatically by the scheduled pipeline monthly-pre-deploy in the release-tools repo. Please ensure that it ran successfully.

Up until Friday, Aug 9

Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
Push any successful deploy to canary into production after some time has passed (preferably 1h).
Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
2. If permission is granted, set the OVERRIDE_PRODUCTION_CHECKS_REASON as a variable in the manual promote job. The value of the variable should be the reason why production checks are being overridden. See gitlab-com-deployer.md#skipping-production-promotion-checks for more info.
3. This will post a comment into this issue and begin the deployment
4. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

Release Preparation

Initial preparation day: Friday, Aug 9

Find the latest sha that made it into production successfully: sha
Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
Manually create the pipeline to update monthly release status to announced.

Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 15th:

/chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 15th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 17.3`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`
  * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1"

Verify that the release information dashboard reflects the accurate status "announced" for the active release version (17.3).
- The updated status can take up to 15 minutes to display on the dashboard once the pipeline to create the metric is finished.

Candidate announcement day: Monday, Aug 12

Log latest auto-deploy branch: 17.3.202408121405
Ensure this build makes it through into production
Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release:

/chatops run notify ":mega: This is the _candidate_ commit to be released on the 15th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/5315e05d527
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 17.3`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`
  * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1"

RC tag day: Tuesday, Aug 13

Determine what is the last auto deploy branch to have deployed to production and add it here: 17.3.202408130206
If you plan to use the latest commit deployed to production for the various components to create the RC, make sure:
- To execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed:
```
/chatops run post_deploy_migrations execute
```
- To verify if there are production incidents blocking deployments
Create a RC version to ensure that the final version builds correctly
```
# In Slack:
/chatops run release tag 17.3.0-rc42
```

This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:

/chatops run release tag 17.3.0-rc42 --gitlab-sha=XXX

This will then use XXX as the SHA to create the GitLab stable branches.

NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

Verify that the CE stable branch contains the right commits
- There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
- The sync commit will have the message "Add latest changes from gitlab-org/gitlab@17-3-stable-ee"

Verify that the pipelines are green

# In Slack:
/chatops run release status 17.3.0-rc42

Verify that the RC has been deployed to the pre environment
- Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
- If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.
Verify that the release information dashboard reflects the accurate status "RC Tagged" for the active release version (17.3).
- The updated status can take up to 15 minutes to display on the dashboard once the RC is tagged.

Notify Engineering Managers and developers that final candidate has been created:

/chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 15th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-3-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 17.3`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`
  * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1"

Tag day: Wednesday, Aug 14

Note: You need to wait for the jobs in the tag_day stage to be successful before tagging

Start the monthly_release_tag_day:start job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3604029
Tag 17.3.0:
```
# In Slack:
/chatops run release tag 17.3.0
```
- Check progress of EE packages build and CE packages build
Validate 17.3.0 has been deployed to the release environment

Instructions for manual deploy

    ```sh
    # In Slack:
    /chatops run deploy 17.3.0-ee.0 release
    ```

Validate 17.3.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

Release day: Thursday, Aug 15

Final release is tagged, so any changes will have to initiate a patch release. Reminder: We have a soft PCL today, coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.

At 13:00 UTC, start the monthly_release_release_day:start job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3604029
At 13:10 UTC:
- ⚠ Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
- Publish the packages via ChatOps. Check the list of pipelines printed at the end of the CI job logs, and make sure the pipelines pass.
```
# In Slack:
/chatops run publish 17.3.0
```
- If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
- At 14:10 UTC:
  - Verify the check-packages job completes successfully on the EE Pipeline
  - Verify the check-packages job completes successfully on the CE Pipeline
  - Start the monthly_release_verify:start job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3604029
  - Ensure the jobs on the pipeline are successfully completed
  - Post an update about the status in #f_upcoming_release
```
:mega: 17.3.0 is published and publicly available
```
  - Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC
  - Start the monthly_release_finalize:start job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/3604029

Release Certification

The release certification process may apply to this release. cc @gitlab-com/gl-security/product-security/federal-application-security

Edited Aug 15, 2024 by Dat Tang