<!--If you make any changes to this template in Release Tools, also make sure to update any existing release issues (if necessary). Similarly, if you update the release issue please consider whether your change needs to go into the template for future release managers - https://gitlab.com/gitlab-org/release-tools/-/tree/master/templates--> ## First steps - [x] Change `#f_upcoming_release` topic with `/topic 19.5.0: <link_to_this_issue>` - [x] Check for any holiday, [Family & Friends day](https://handbook.gitlab.com/handbook/company/family-and-friends-day/#upcoming-family-and-friends-days) or [planned PCLs](https://gitlab.com/gitlab-com/gl-infra/change-lock) scheduled for this release: - [ ] Adjust the [release preparation](#release-preparation) days if necessary - [ ] If there are Family and Friends day add notes for pausing deployments before the day starts and unpausing them before the next business day, see the [auto-deploy documentation](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#do-we-deploy-on-weekends-or-holidays) for details. - [ ] Update this issue with two planned dates for [recurring Staging rollback practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md) and a planned date for [Hot Patching Production practice](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/post-deployment-patches.md#monthly-release-hot-patching-production-practice). Consider spreading these across timezones to share the knowledge. - [ ] Set Due Date for this Issue to the first practice session - [ ] Check for any [deprecations](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/deprecation-evaluation.md) ## Optional: First Staging Rollback Practice \[Skipped\] > [!note] > > If the release manager is proficient with the rollback procedure or already performed at least one rollback during the current rotation, they can skip the practice. **Date to be Completed:** **Skipped.** We did a rollback on 2026-09-18, on the first day of this shift: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/6260042 So, we don't need to practice it. - [ ] [Perform Staging Rollback Practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md) - [ ] Set Due date for this issue to the date of the next practice session. If none, set it to the release date. ## Optional: Second Staging Rollback Practice \[Skipped\] > [!note] > > If the release manager is proficient with the rollback procedure or already performed at least one rollback during the current rotation, they can skip the practice. **Date to be Completed:** **Skipped.** - [ ] [Perform Staging Rollback Practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md) - [ ] Set Due date for this issue to the date of the next practice session. If none, set it to the release date. ## Hot patching production practice **Date to be Completed: 2026-09-29** _The work here is to be done by the next release managers. Please tag them to the item below._ - [x] `@madelacruz` - [x] [Perform hot patching Practice](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/post-deployment-patches.md#monthly-release-hot-patching-production-practice) - [x] Hot Patching practice is documented: https://gitlab.com/gitlab-org/release/tasks/-/work_items/27879#note_3920290600 - [x] Set Due date for this issue to the date of the next practice session. If none, set it to the release date. ## Up until Tuesday, 2026-10-13 - Ensure any deploys that do not make it to canary are investigated. [Disable canary if necessary](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/canary.md#how-to-stop-all-production-traffic-to-canary). - Push any successful deploy to canary into production after some time has passed (preferably 1h). - Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call. 1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer 2. If permission is granted, set the `OVERRIDE_PRODUCTION_CHECKS_REASON` as a variable in the manual `promote` job. The value of the variable should be the reason why production checks are being overridden. See [gitlab-com-deployer.md#skipping-production-promotion-checks](https://gitlab.com/gitlab-org/release/docs/-/blob/6d9db5984c5de288d1be54e202d0253fedb50b82/general/deploy/gitlab-com-deployer.md#skipping-production-promotion-checks) for more info. 3. This will post a comment into this issue and begin the deployment 4. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes ## Release Preparation ### Initial RC: Tuesday, 2026-10-13 - [ ] Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: ```text /chatops gitlab run post_deploy_migrations execute ``` - [ ] If there was an [Internal release](https://gitlab.com/gitlab-org/release/tasks/-/issues/?state=all&label_name%5B%5D=internal%20releases) after the last patch release and it includes a security fix: - [ ] Select a commit before the internal release merge request for the next tagging step: - [ ] Link the Internal release issue: - [ ] Start the `monthly_release_initial_rc:start` job of the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298 - If a specific commit is needed, submit it as a job variable (`STABLE_BRANCH_SOURCE_COMMITS_INPUT`) in the `monthly_release_initial_rc:start` job ``` STABLE_BRANCH_SOURCE_COMMITS_INPUT='gitaly=123abc,gitlab=456def,gitlab_pages=678cba' ``` > [!note] > > Each commit must be from the project's default branch (`master` for most components, not from an `auto-deploy` branch). Using a commit from an `auto-deploy` branch will cause the tagging job to fail before any component is tagged. See [documentation](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/monthly/release-candidates.md#default-creating-a-rc-via-monthly-release-pipeline) for additional details - If no commit is provided, the tooling will use the latest commit deployed to production for the various components that we release. See the [release candidates doc](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/monthly/release-candidates.md#default-creating-a-rc-via-monthly-release-pipeline) for detailed guidance. - When the `monthly_release_initial_rc:start` job is started, the initial RC will be tagged automatically. - 120 minutes after the initial RC is tagged, jobs to verify the RC pipelines and the deployment to preprod will be automatically triggered. - [ ] Verify that the [release information dashboard](https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1) reflects the accurate status "closed" for the active release version (19.5). - The updated status can take up to 15 minutes to display on the dashboard once the RC is tagged. - [ ] Notify Engineering Managers and developers that the initial release candidate has been created: ``` /chatops gitlab run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 15th. https://gitlab.com/gitlab-org/security/gitlab/-/commits/19-5-stable-ee You can check if an MR made the cut by using the following ChatOps command: `/chatops gitlab run release check [MR_URL] 19.5` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases` * When is the next Release? Check it on the 'Release Information' dashboard: https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1" ``` Note: If required, [additional release candidates can be manually tagged](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/monthly/release-candidates.md#tagging-manually-using-chatops) before the final release candidate. ### Final RC: Tuesday, 2026-10-13 - [ ] Start the `monthly_release_final_rc:start` job of the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298. - [ ] Ensure all the jobs in the `monthly_release_final_rc` stage succeed. Past this point, no new code can be added to the release that was not included in the final RC. ### Tag day: Wednesday, 2026-10-14 (3rd Wednesday of the month) - [ ] Start the `monthly_release_tag_day:prepare` job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298 - Jobs to verify the integrity of the stable branches (`monthly_release_tag_day:ensure_stable_branches_green`) and the mirror status (`monthly_release_tag_day:mirror_status`) will start automatically. - [ ] Once the above jobs are green, trigger the `monthly_release_tag_day:start_tag` job: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298 - This job will tag the final `19.5.0` version. - [ ] The `monthly_release_tag_day:check_omnibus_packages_tagging` job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298 will be triggered 90 minutes after tagging completes. However, if possible, you can start it earlier. In case the job fails, troubleshoot the issue, and rerun the job. - If this job fails due to the `check-packages-functionality` downstream job, follow this [runbook](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/check-packages-functionality-failure.md) - [ ] Ensure `monthly_release_tag_day:check_cng_packages_tagging` succeeds. If not, follow the [CNG troubleshoot docs](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/manual-cng-verification.md) and rerun the job. - [ ] Ensure `monthly_release_tag_day:check_helm_packages_tagging` succeeds. If not, follow the [Helm chart troubleshoot docs](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/charts/README.md) and rerun the job. - [ ] After tagging, the [GitLab QA update scenario](https://handbook.gitlab.com/handbook/engineering/testing/test-coverage/#gitlab-qa-update-scenario) runs. Corroborate if the QA was successful by verifying that the `monthly_release:update_paths` stage succeeded. - A Slack notification with `Update paths QA` will be sent to `#f_upcoming_release`. ### Release day: Thursday, 2026-10-15 (3rd Thursday of the month) Final release is tagged, so any changes will have to initiate a patch release. [Reminder: We have a soft PCL today](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#soft-pcl), coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available. - [ ] At 13:00 UTC, start the `monthly_release_release_day:start` job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298. An announcement will be sent and the publish will happen 10 minutes later automatically. In case you want to publish the release immediately, skip the delay time of the job `monthly_release_release_day:publish`. - :warning: **Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the** [**messaging lead**](https://handbook.gitlab.com/handbook/marketing/technical-writing/#schedule) **of the release post. Mind that you don't need their approval if you're on time** :warning: - If anything goes wrong and the release is delayed, ping the [release post manager](https://handbook.gitlab.com/handbook/marketing/technical-writing/#schedule) on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too * [ ] * At 14:10 UTC: - [ ] Start the `monthly_release_verify:start` job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298 - [ ] Make sure the `monthly_release:verify` stage has completed successfully. - If there are Omnibus failures, follow the [Omnibus troubleshoot docs](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/omnibus-packages/manual-omnibus-verification.md) and rerun the job. - If there are CNG failures, follow the [CNG troubleshoot docs](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/manual-cng-verification.md) and rerun the job. - If there are Helm failures, follow the [Helm chart troubleshoot docs](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/charts/README.md) and rerun the job. - [ ] Post an update about the status in `#f_upcoming_release` ``` :mega: 19.5.0 is published and publicly available ``` * [ ] Once all packages are available publicly and GitLab.com is up and running on the release version, ping the [release post manager](https://handbook.gitlab.com/handbook/marketing/technical-writing/#schedule) on Slack ([#release-post channel](https://gitlab.enterprise.slack.com/archives/C3TRESYPJ)) to give them a go to merge the release post at \~14:20 UTC, so that it will be live at 15:00 UTC * [ ] Start the `monthly_release_finalize:start` job in the monthly release pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6258298 ## Release Certification The [release certification process](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/release-certification.html) may apply to this release. cc @gitlab-com/gl-security/product-security/federal-application-security