# Production Change - Criticality 2 ~C2 ## Change Summary This production issue is to be used for Gamedays as well as recovery in case of a zonal outage. It outlines the steps to be followed when testing traffic shifts due to zonal outages. Hopefully corrective actions from testing will help us build new steps to take during a real outage. ### Gameday execution roles and details | Role | Assignee | |------|----------| | Change Technician | @sun_lee | | Change Reviewer | @haseeb_ahmed | - **Services Impacted** - ~&quot;Service::HAProxy&quot; - **Time tracking** - 90 minutes - **Downtime Component** - 30 minutes - **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-08-17 00:00 Restricting traffic to operating in two remaining zones due to a zonal outage in `us-east1-b` . <!--e.g Restricting traffic to operating in two remaining zones due to a zonal outage in us-east1-d--> ## Important Notes - **Node Numbering Strategy:** Use node numbers starting at 101 (e.g., 101, 102, 103) for additional nodes created during DR exercises to clearly distinguish them from regular production nodes. - **Configuration Approach:** Use nodes_overrides in Terraform for adding new nodes rather than modifying nodes_count to avoid disrupting existing nodes. - **Bootstrap Timing:** New node bootstrapping can take up to 30 minutes. Plan accordingly and use the validation steps to confirm completion. - **YubiKey Touch Prompts:** Several commands in this gameday (e.g. `knife`, `./bin/disable-server`, `./bin/enable-server`, `./bin/manage-gcp-lb-haproxy`, `./bin/get-server-state`) will require a YubiKey touch to authenticate. Unlike interactive sessions, these gameday commands do **not** show a desktop pop-up prompt — the command will just appear to hang. Watch for the **flashing LED on your YubiKey** as the reliable indicator that a touch is required. ## \[**For Gamedays only**\] Preparation Tasks ### One week before the gameday 1. [x] Update the title of this issue to be `FYXX-QX HAProxy DR Gameday (outage in <availability-zone>)` - **Zone rotation:** alternate the affected zone each quarter so we exercise different zones over time — odd quarters (Q1, Q3) use `us-east1-b`, even quarters (Q2, Q4) use `us-east1-c`. 2. [x] Make an announcement on the [#f_gamedays](https://gitlab.enterprise.slack.com/archives/C07PV3F6J1W) Slack channel with this template: ```code Next week on [DATE & TIME] we will be executing a Traffic Routing game day. The process will involve moving traffic away from a single zone in `gstg` to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets set by the [DR working group](https://handbook.gitlab.com/handbook/company/working-groups/disaster-recovery/) for GitLab.com. See [URL TO THIS ISSUE] ``` Then cross post the message to the following channels: - [#g_production_engineering](https://gitlab.enterprise.slack.com/archives/C03QC5KNW5N) - [#staging](https://gitlab.enterprise.slack.com/archives/CLM200USV) (if applicable) - [#production](https://gitlab.enterprise.slack.com/archives/C101F3796) (if applicable) 3. [x] Create an MR to reconfigure the regional cluster to exclude the affected zone by setting `regional_cluster_zones` in Terraform to a list of zones that are not impacted - Create the MR to update the `regional_cluster_zones`. While _emulating_ a zonal outage make sure to create replacement nodes using the `nodes_overrides` parameter rather than modifying `nodes_count`. Refer to this [example MR](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/12257). - Reference this approach for additional nodes: use `nodes_overrides` with node numbers starting at 101 (e.g., 101, 102, 103) to clearly distinguish them from existing nodes. - Link to the MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15014 ```bash # command to list all haproxy vms in a specific zone $ gcloud --project gitlab-staging-1 compute instances list \ --filter="labels.gitlab_com_service=haproxy AND zone:us-east1-c" \ --format="table(name,zone)" NAME ZONE haproxy-ci-01-lb-gstg us-east1-c haproxy-main-02-lb-gstg us-east1-c haproxy-pages-02-lb-gstg us-east1-c haproxy-registry-02-lb-gstg us-east1-c ``` - Replace `MR_LINK` with this MR in the template 4. [x] Get the MR approved 5. [x] Mention the release managers on the Slack announcement by mentioning `@release-managers` and await their approval. 6. [x] Request approval from the Infrastructure manager by mentioning `@gitlab-org/saas-platforms/change-review-leadership`, wait for approval and confirm by the ~platform_leadership_approved label. ### Just before the gameday begins 5. [x] Before commencing the change, notify the EOC and release managers on _Slack_ with the following template and wait for their acknoledgement and approval ```code @release-managers or @sre-oncall [LINK_TO_THIS_CR] is scheduled for execution today at [TIME]. We will be diverting traffic away from a single zone ([NAME_OF_ZONE]) in `gstg` to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets. Kindly review and approve the CR. ``` ## Detailed steps for the change ### Change Steps - steps to take to execute the change #### Execution 1. [ ] If you are conducting a practice (Gameday) run of this, consider starting a recording of the process now. 2. [x] Note the start time in UTC in a comment to record this process duration. 3. [x] Set label ~&quot;change::in-progress&quot; `/label ~change::in-progress` 4. [x] Run `/atlantis apply` on https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15014 {+ +}which was created earlier 5. [x] Confirm that the haproxy VMs are added to the tcp load balancers, and they are registered in Chef ```bash cd chef-repo # check Chef status bundle exec knife search -i node "name:haproxy* AND chef_environment:gstg" | sort # check LB status # will prompt for an environment, then a zone. Select "list", and the values correlating to the zone that we are adding capacity to ./bin/manage-gcp-lb-haproxy ``` 6. [x] Disable the HAproxy servers: ```bash cd chef-repo ./bin/disable-server gstg <impacted zone> ``` 7. [x] Validate that all servers in the affected zone have their state set to `MAINT` by running `./bin/get-server-state` in `chef-repo` ```bash cd chef-repo ./bin/get-server-state gstg ``` 8. [x] Remove the HAProxy instances from the GCP load balancers ```bash cd chef-repo # will prompt for an environment, then a zone. Select the values correlating to the zone we are removing traffic from. ./bin/manage-gcp-lb-haproxy ``` 9. [x] Note the conclusion time in UTC in a comment to record this process duration. #### Validation Once traffic is restricted to our remaining two zones, let's identify the impact and look for problems. - [x] Do we see a drop in CPU usage in one zone cluster? [GSTG Per Cluster CPU Usage](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22pum%22:%7B%22datasource%22:%22mimir-gitlab-gstg%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%20by%20%28cluster%29%20%28rate%28container_cpu_usage_seconds_total:labeled%7Benv%3D%5C%22gstg%5C%22,namespace%3D%5C%22gitlab%5C%22%7D%5B5m%5D%29%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gstg%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) (it may be not noticable when validating on the `gstg` environment due to the low traffic) - [x] Do we see a drop in HPA targets in one zone cluster? [GSTG Per Cluster HPA Target](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22pum%22:%7B%22datasource%22:%22mimir-gitlab-gstg%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%20by%20%28cluster%29%20%28kube_horizontalpodautoscaler_status_current_replicas%7Benv%3D%5C%22gstg%5C%22,namespace%3D%5C%22gitlab%5C%22%7D%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gstg%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) (it may be not noticable when validating on the `gstg` environment due to the low traffic) - [x] [Examine GSTG Rails logs for errors](https://nonprod-log.gitlab.net/app/r/s/005mL) - [x] [Examine frontend dashboard for GSTG](https://dashboards.gitlab.net/d/frontend-main/frontend3a-overview?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gstg&var-environment=gstg&var-stage=main&from=now-1h&to=now) - [x] [Examine the connected peers changed with the new introduced nodes](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22yju%22:%7B%22datasource%22:%22mimir-gitlab-gstg%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%20%28haproxy_process_connected_peers%29%20by%20%28fqdn%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gstg%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-30m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) #### Wrapping up and cleanup 1. [x] Compile the real time measurement for all the new HAProxy nodes by running the [script](https://gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/find-bootstrap-duration.sh?ref_type=heads) and comment with the output on this issue: ```bash # Make sure to filter for the correct environment (gstg in this case) for node in `knife node list | grep gstg | grep haproxy | grep 101`; do ./runbooks/scripts/find-bootstrap-duration.sh $node ; done ``` 2. [x] Re-enable the zonal GKE backend cluster in HAProxy ```bash cd chef-repo ./bin/enable-server gstg <impacted zone> ``` 3. [x] Validate that all servers in the affected zone have their state set to `UP` by running `./bin/get-server-state` in `chef-repo` ```bash cd chef-repo ./bin/get-server-state gstg ``` 4. [x] Open an revert MR https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15091 and run `/atlantis apply` when it is approved 5. [x] Run an full [Terraform plan/apply for GSTG](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/pipelines/new?ref=main&var%5BENV%5D=gstg&var%5BAPPLY_FULL%5D=true) from the main branch. This needs to be done to ensure the HAProxy nodes are added back to the TCP LBs. 6. [x] **IMPORTANT** Confirm that the HAProxy nodes are added back to the TCP LBs that were removed earlier ```bash cd chef-repo # check LB status # will prompt for an environment, then a zone. Select "list", and the values correlating to the zone that was affected ./bin/manage-gcp-lb-haproxy ``` 7. [x] Set label ~&quot;change::complete&quot; `/label ~change::complete` 8. [x] Notify the `@release-managers` and `@sre-oncall` that the exercise is complete. 9. [x] Compile the real time measurement of this process and update the [Recovery Measrements Runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/disaster-recovery/recovery-measurements.md?ref_type=heads). ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change #### _It is estimated that this will take 5m to complete_ - [ ] [Re-enable HAProxy with Terraform](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/pipelines/new?ref=main&var%5BENV%5D=gstg&var%5BAPPLY_FULL%5D=true). This needs to be done to ensure the HAProxy nodes are added back to the TCP LBs. - [ ] Re-enable HAProxy ```bash cd chef-repo ./bin/enable-server gstg <impacted zone> ``` - [ ] Revert {+MR_LINK +}and run `/atlantis apply` when it is approved - [ ] Set label ~&quot;change::complete&quot; `/label ~change::aborted` - [ ] Notify the `@release-managers` and `@sre-oncall` that the exercise has been aborted. ## Change Reviewer checklist <!--To be filled out by the reviewer.--> ~C4 ~C3 ~C2 ~C1: - [ ] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed upon with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~&quot;blocks deployments&quot; and/or ~&quot;blocks feature-flags&quot; are applied as necessary. ## Change Technician checklist <!--To find out who is on-call, use the `@sre-oncall` handle in slack--> - [ ] Check if all items below are complete: - The [change plan](#detailed-steps-for-the-change) is technically accurate. - This Change Issue is linked to the appropriate Issue and/or Epic - Change has been tested in staging and results are noted in a comment on this issue. - A dry-run has been conducted and results are noted in a comment on this issue. - The change execution window respects the [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl). - For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar. - For ~C1 and ~C2 change issues, the SRE on-call has been informed before the change is rolled out. (In the #production channel, mention `@sre-oncall` and this issue and await their acknowledgement.) - For ~C1 and ~C2 change issues, the SRE on-call provided approval with the ~eoc_approved label on the issue. - For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - Release managers have been informed prior to any C1, C2, or ~&quot;blocks deployments&quot; change being rolled out. (In the #production channel, mention `@release-managers` and this issue and await their acknowledgement.) - There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name%5B%5D=Incident%3A%3AActive) that are ~severity::1 or ~severity::2 - If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.