<!--Please review https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/ for the most recent information on our change plans and execution policies.--> # Production Change ## Change Summary - Related to: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2231 - Tracking issue: https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22422 - Follows: https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22421 (Omnibus and CNG packages verified on Rails 8.0.5.1) - Previous iteration: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20789 (Rails 7.2) We will test deploying a Rails 8.0 package to both canary environments (`gstg-cny` and `gprd-cny`) to catch issues that our pipelines did not. Production main stages are not affected: the test package is not promoted beyond `gprd-cny`. ## Change Details <!--To automatically add your change to the GitLab Production calendar update the following fields: - Time tracking - Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) Bot: https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler--> 1. **Services Impacted** - GitLab.com 2. **Change Technician** - `@anganga` (EMEA). If the CR runs past the EMEA shift, I will hand over to @jennykim-gitlab (AMER) 3. **Change Reviewer** - @jennykim-gitlab, @dat.tang.gitlab 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-10-02 10:00 5. **Time tracking** - 8 hours 6. **Downtime Component** - none > [!note] > > Bakes end after about 2 hours **or** when the Rails dev DRI gives the green light, whichever is sooner. The Rails 7.2 test rollout blocked deployments for 16 hours against 8 planned. ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~&quot;change::scheduled&quot; ### Change Reviewer checklist <!--To be filled out by the reviewer.--> ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~&quot;blocks deployments&quot; and/or ~&quot;blocks feature-flags&quot; are applied as necessary. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [ ] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [ ] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~&quot;blocks deployments&quot; change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps > [!note] > > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are listed below. - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out and has provided approval with the ~eoc_approved label on the issue. - [ ] Check for [active \~severity::1 / \~severity::2 incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) before requesting approval. If one overlaps with this change's blast radius, reschedule. If it does not overlap, include a brief risk assessment in the `@sre-oncall` approval request; if the EOC is unavailable or does not approve, reschedule. - [ ] For ~C1, ~C2, or ~&quot;blocks deployments&quot; change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. - [x] Have an AMD64 host ready for verification (the toolbox image was only checked on AMD64; Omnibus now also publishes an `-arm64` image, see `Docker-arm64` job). ```shell gcloud compute instances create rails8-verify \ --project=<sandbox-project> \ --zone=us-central1-a \ --machine-type=e2-standard-4 \ --image-family=debian-12 --image-project=debian-cloud \ --boot-disk-size=30GB --boot-disk-type=pd-balanced ``` - If a zone is out, try another. - 30GB: measured on 2026-09-30, both images take 8.2 GB and the disk showed 12G used after pulling them. ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - 480 mins #### Setting the stage - [x] Set label ~&quot;change::in-progress&quot; `/label ~change::in-progress` - [x] Pause auto-deploy: `/chatops gitlab run auto_deploy pause` * [ ] * If it fails with a 403 (https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22434), pause by hand: - [x] Deactivate the `auto_deploy: Deploy latest available package` [schedule](https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules) - [x] Turn on the `do_not_auto_create_new_deployment_pipelines` [feature flag](https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags) * [x] Confirm: the flag is `true` and the schedule is inactive ```shell glab api --hostname ops.gitlab.net "projects/gitlab-org%2Frelease%2Ftools/feature_flags/do_not_auto_create_new_deployment_pipelines" | jq .active ``` - [x] Make sure the job `deploy:start-new-pipeline` of the latest auto-deploy pipeline doesn't run (cancel it if it hasn't started), so it doesn't create a new deployment pipeline: * https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6335438 #### Preparing the auto-deploy package - [x] Add an empty commit to the latest Omnibus auto-deploy branch in `gitlab-org/security/omnibus-gitlab` (branches are cut hourly at :05; pick the branch after the :05 cut so it isn't superseded mid-step): ```shell git clone --depth 1 --branch 19-5-auto-deploy-2026100211 git@gitlab.com:gitlab-org/security/omnibus-gitlab.git git commit --allow-empty -m "Empty commit to trigger a new auto-deploy pkg" git push origin 19-5-auto-deploy-2026100211 ``` - Branch: `19-5-auto-deploy-2026100211` - Commit: https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/commit/dcd0c4cb3eba0dc9015f0ac6e32b7d2c3b6dddc7 - [x] Trigger creation of a new package by running the `auto_build: Create new package` [scheduled pipeline](https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules). Find the tag on [Omnibus](https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines?scope=tags), [CNG](https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines?scope=tags) or [auto-deploy view](https://auto-deploy-view-e121e0.gitlab.io/). The tag ends in your commit's short SHA. - Omnibus tag: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/commits/19.5.202610021113+0b8f815ca85.dcd0c4cb3eb - CNG tag (same, without the Omnibus SHA suffix): https://dev.gitlab.org/gitlab/charts/components/images/-/commits/19.5.202610021113+0b8f815ca85 - [x] :warning: Cancel the pipelines created automatically for that tag (they build **without** `Gemfile.next`): - [x] Omnibus: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/464980 - [x] CNG: https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/464982 - [x] Manually start new packager pipelines **on the same tags** with `GITLAB_BUNDLE_GEMFILE=Gemfile.next`. Change the ref to the tag before clicking Run: - [x] [Omnibus](https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/new?var%5BGITLAB_BUNDLE_GEMFILE%5D=Gemfile.next): https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/464987 - [x] [CNG](https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/new?var%5BGITLAB_BUNDLE_GEMFILE%5D=Gemfile.next): https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/464989 - [x] Confirm the variable reached both pipelines (only variables set at start time are returned): ```shell glab api --hostname dev.gitlab.org "projects/gitlab%2Fomnibus-gitlab/pipelines/464987/variables" glab api --hostname dev.gitlab.org "projects/gitlab%2Fcharts%2Fcomponents%2Fimages/pipelines/464989/variables" ``` * Expected duration: Omnibus \~45 min, CNG \~65 min (22421, 2026-09-30). #### Verify > [!note] > > Step 1 (install Docker and log in to the dev registry) doesn't need the packages, so do it while the Omnibus and CNG pipelines are running. ##### 1. Install Docker and log in to the dev registry ```shell gcloud compute ssh rails8-verify --project=<sandbox-project> --zone=us-central1-a sudo apt-get update -qq && sudo apt-get install -y -qq docker.io sudo docker login dev.gitlab.org:5005 -u <dev.gitlab.org username> ``` The password must be a **dev.gitlab.org** personal access token with `read_registry`. A gitlab.com token is rejected (`HTTP Basic: Access denied`). ##### 2. Check the Rails version Image names come from the `Docker` job (Omnibus) and the `gitlab-toolbox-ee` job (CNG) in the pipelines above. - [x] Omnibus: ```shell sudo docker run -it --rm dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:19-5-202610021113-0b8f815ca85-dcd0c4cb3eb /bin/bash # inside the container # start the service supervisor first; with /bin/bash as the command the image's entrypoint # doesn't run, and reconfigure hangs at "wait for logrotate service socket" /opt/gitlab/embedded/bin/runsvdir-start & gitlab-ctl reconfigure gitlab-rails version # expect Rails 8.0.x ``` - [x] CNG: ```shell sudo docker run -it --rm dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-toolbox-ee:19-5-202610021113-0b8f815ca85 /srv/gitlab/bin/rails --version # expect Rails 8.0.x ``` - [x] Paste both outputs here. ```shell gitlab Reconfigured! root@e34bf2c1ae11:/# gitlab-rails version Rails 8.0.5.1 root@e34bf2c1ae11:/# Digest: sha256:5b9e761f38dcfa3617f84fb68b3077f31a309dcf093611d10edb46ef52876e3f Status: Downloaded newer image for dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-toolbox-ee:19-5-202610021113-0b8f815ca85 Begin parsing .erb templates from /srv/gitlab/config Begin parsing .tpl templates from /srv/gitlab/config Attempting to run '/srv/gitlab/bin/rails --version' as a main process Rails 8.0.5.1 ``` ##### 3. Clean up - [x] `gcloud compute instances delete rails8-verify --project=<sandbox-project> --zone=us-central1-a` - [x] Revoke the dev.gitlab.org token if nothing else uses it. #### Rollout to gstg-cny - [x] :lock: `/chatops gitlab run deploy lock gprd-cny` to prevent deploying beyond `gstg-cny` until we get the green light from the Rails dev DRI (`@igor.drozdov`). - [x] `/chatops gitlab run auto_deploy pipeline 19.5.202610021113` with the package version, once both packager pipelines have succeeded. Before both finish it fails with `UnsuitableVersionError`. `--force` starts the deploy pipeline straight away (it still waits for the packages in `wait:omnibus` / `wait:cng`), but it also skips the Sequencer check, so use it only when no other deployment to canary is in progress. * [x] Pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/6336354 - [x] :warning: Cancel the `validate_ownership:gstg-cny` job to prevent a deployment to gprd-cny from starting. - [x] :eyes: Keep an eye on the auto-deploy pipeline and do not allow it to be deployed beyond staging-canary. - [ ] :mega: Notify Slack channels `#development`, `#backend`, `#frontend` and `#staging-ref` that the deployment has started. - [ ] :mega: Once the deployment to `gstg-cny` is done, notify Rails dev DRI (`@igor.drozdov`) so he can start monitoring. #### Baking in gstg-cny - [x] :hourglass_flowing_sand: Bake in `gstg-cny` until monitoring is over: about 2 hours, or sooner with the Rails dev DRI's green light. - [x] Get the green light from engineers helping with monitoring to proceed with deployment in gprd-cny. #### Rollout to gprd-cny - [x] :unlock: Unlock `gprd-cny`: `/chatops gitlab run deploy unlock gprd-cny` - [x] :mega: Let `@sre-oncall` and `@release-managers` know in `#production` that the test package is going to be rolled out to `gprd-cny`. - [x] Restart the previously cancelled `validate_ownership:gstg-cny` job to proceed with the deployment to `gprd-cny`. - [x] :warning: Cancel/unschedule the following jobs to ensure that **this deployment does not get deployed to the main stages**: - [x] `validate_ownership:gprd-cny` - [ ] `baking_time` - [x] :mega: Notify rails dev DRI (`@igor.drozdov`) and engineers helping with monitoring when the deployment to `gprd-cny` is done, so they can start monitoring. #### Baking in gprd-cny - [x] Bake in `gprd-cny` until monitoring is over: about 2 hours, or sooner with the Rails dev DRI's green light. - [x] Green light from engineers helping with monitoring. #### Continuing with auto-deployment The test packages were built with a pipeline-level variable, so every other auto-deploy package is still built with the current Rails version. We will **not** promote the Rails 8.0 test package to staging or production. - [x] Wait for an auto-deploy package newer than the test package, then `/chatops gitlab run auto_deploy pipeline <newer version>` so the next deployment moves canary off the test package. - [x] :mega: Let `@sre-oncall` and `@release-managers` know in `#production` that the CR is completed and auto deployments will resume. - [x] Resume auto-deploy: `/chatops gitlab run auto_deploy unpause` - If it fails with a 403 (22434), resume by hand: - [x] Turn off the `do_not_auto_create_new_deployment_pipelines` [feature flag](https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags) - [x] Re-activate the `auto_deploy: Deploy latest available package` [schedule](https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules) - [x] Set label ~&quot;change::complete&quot; `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 3 minutes At any point during the test rollout to `gstg-cny`, if it is deemed unsuccessful by the engineers helping with monitoring the rollout, we will not deploy to `gprd-cny`. This package is not meant to be deployed beyond `gprd-cny`. Whether or not the test rollout to `gprd-cny` is successful, we will continue the auto-deployment process with packages built with the current Rails version. We will **not be rolling back** the deployment of the Rails 8.0 package; the next auto-deploy package replaces it on canary. - [ ] In case of a broken production canary deploy, [drain the canary traffic](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/canary.md#how-to-stop-all-production-traffic-to-canary) in `#production`: ```text /chatops gitlab run canary --disable --production ``` - [ ] Once a successful production canary deployment of a newer package has completed, re-enable canary traffic: ```text /chatops gitlab run canary --enable --production ``` - [ ] Resume auto-deploy as in [Continuing with auto-deployment](#continuing-with-auto-deployment), including the manual fallback. - [ ] Set label ~&quot;change::aborted&quot; `/label ~change::aborted` ## Monitoring ### Key metrics to observe * Dashboards/metrics: * Monitor the following dashboards for unhealthy dip in service health for the environment/cluster that is being rolled out. * [Deployment health](https://dashboards.gitlab.net/d/delivery-deployment_health/delivery-deployment-health?orgId=1), configurable with environment, stage, and type/service * [Kubernetes compute resource/cluster health](https://dashboards.gitlab.net/d/kubernetes-resources-cluster/kubernetes-compute-resources-cluster?orgId=1&refresh=5m), configurable with clusters * [Kubernetes compute resource/pods health](https://dashboards.gitlab.net/d/kubernetes-resources-namespace/kubernetes-compute-resources-namespace-pods?orgId=1&refresh=5m), configurable with clusters and namespace * [Kubernetes networking](https://dashboards.gitlab.net/d/kubernetes-cluster-total/kubernetes-networking-cluster?orgId=1&refresh=5m), configurable with clusters * Per-service dashboards (change `env` and `stage` to toggle between `gstg`/`gprd` and `main`/`cny`): * `api` ([overview](https://dashboards.gitlab.net/d/api-main/api-overview?orgId=1&from=now-1h&to=now), [containers](https://dashboards.gitlab.net/d/api-kube-containers/api-kube-containers-detail?from=now-1h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&orgId=1)) * `web` ([overview](https://dashboards.gitlab.net/d/web-main/web-overview?orgId=1&from=now-1h&to=now), [containers](https://dashboards.gitlab.net/d/web-kube-containers/web-kube-containers-detail?from=now-1h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&orgId=1)) * `websockets` ([overview](https://dashboards.gitlab.net/d/websockets-main/websockets-overview?orgId=1&from=now-1h&to=now), [containers](https://dashboards.gitlab.net/d/websockets-kube-containers/websockets-kube-containers-detail?from=now-1h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&orgId=1)) * `git` ([overview](https://dashboards.gitlab.net/d/git-main/git-overview?orgId=1&from=now-1h&to=now), [containers](https://dashboards.gitlab.net/d/git-kube-containers/git-kube-containers-detail?from=now-1h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&orgId=1)) * `sidekiq` ([overview](https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1&from=now-1h&to=now), [containers](https://dashboards.gitlab.net/d/sidekiq-kube-containers/sidekiq-kube-containers-detail?from=now-1h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&orgId=1)) * Kibana - Puma (edit `json.type` to filter by service, `json.stage` for `cny` vs `main`) * [Production 5xx responses](https://log.gprd.gitlab.net/goto/e0d9a290-b8c9-11ed-85ed-e7557b0a598c) * [Staging 5xx responses](https://nonprod-log.gitlab.net/goto/040ba510-b8ca-11ed-9af2-6131f0ee4ce6) * Kibana - Sidekiq (edit `json.shard` to switch between job types) * [Failed production jobs](https://log.gprd.gitlab.net/goto/89320700-b813-11ed-9f43-e3784d7fe3ca) * [Failed staging jobs](https://nonprod-log.gitlab.net/goto/e2744200-b814-11ed-9af2-6131f0ee4ce6) * Sentry * [Production overview](https://sentry.gitlab.net/gitlab/gitlabcom/dashboard/?statsPeriod=1h) * [Staging overview](https://sentry.gitlab.net/gitlab/staginggitlabcom/dashboard/?statsPeriod=1h) * QA runs can be observed via Slack: * `#announcements` - Besides QA messages, multiple messages are sent to this channel to account for the different deployments. * QA slack channels - a failure on gstg and gstg-cny is posted in `#qa-staging`, a failure on gprd-cny and gprd in `#qa-production`. * Dealing with deploy failures: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/failures.md