# Production Change ## Change Summary This is a retry of [production#21809](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21809), which was rolled back on 2026-05-06 after the `canary_api` load balancer SLI peaked at \~100% errors and triggered [INC-9883](https://app.incident.io/gitlab/incidents/9883). The root cause has since been identified in [production-engineering#28902 (closed)](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28902) and is corrected in this CR. The goal remains unchanged: route `canary_api` traffic on GPRD away from NGINX and directly to the Kubernetes `gitlab-webservice-api` Service (`10.216.8.13:8181`), as part of [#28303](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28303). ## Change Details 1. **Services Impacted** - ~"Service::API" ~"Service::NGINX" 2. **Change Technician** - @haseeb_ahmed @thisisshreya 3. **Change Reviewer** - @cmcfarland 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-05-18 13:30 5. **Time tracking** - 180 minutes 6. **Downtime Component** - none ## What was wrong last time In [`chef-repo!7254`](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7254), the canary_api role JSON had `"ssl_verify": ""`. The `build_ssl_opts` helper in the `gitlab-haproxy` cookbook ([`libraries/template_helpers.rb`](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/blob/master/libraries/template_helpers.rb#L78)) treats `""` not as "no-ssl" but as "fall through to the fallback backend's value". The fallback for `canary_api` is the main `api` pool, whose `ssl_verify` at the time of the CR was not `no-ssl` — so the rendered `default-server` line ended up as `ssl verify none`. Consequences: - HAProxy attempted TLS handshakes against `10.216.8.13:8181`, which is plaintext HTTP (Puma directly behind the K8s Service, with no SSL terminator in front). - The intermediate fix landed mid-incident ([`chef-repo!7286`](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7286)) explicitly set `check_opts: "no-check-ssl port 8181"`, which fixed the **health check** path and brought the backend back UP — but **real traffic** was still being sent over TLS to the plaintext port, producing the `503 SC-- retries=3 tc=-1` clean TCP refusals observed in BigQuery (\~225k requests). - GSTG did not hit this because its config rendered `default-server no-ssl` correctly — there, both the main `api` and `canary_api` pools had been moved off NGINX in the same change, so `build_ssl_opts`'s fallback chain resolved correctly. Full investigation: [production-engineering#28902 (closed)](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28902). ## What changed in this retry 1. **`ssl_verify` is set explicitly to `"no-ssl"`** in the canary_api role JSON, instead of `""`. This makes the rendered HAProxy config match staging > [!important] > > If your change involves scheduled maintenance, add a step to set and [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) per our runbooks. This will make sure SLA calculations adjust for the maintenance period. > [!warning] > > **Server state persistence window:** During this change, HAProxy server state persistence will be temporarily disabled. While persistence is disabled, any HAProxy reload/restart will reset MAINT/DRAIN state — meaning a drained canary backend will become undrained. This will be communicated to release managers as part of this C1 CR, and **the window between disabling and re-enabling persistence must be kept as short as possible**. See [context](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28303#note_3128214144). > [!note] > > This CR only moves `canary_api` traffic off NGINX. The main `api` backend stays on the NGINX path and will be migrated in a separate follow-up CR. ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl). - [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [x] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps - [x] Prepare and merge [MR](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/460) in [`gitlab-haproxy`](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy) to extend the retry/timeout parity config to `canary_api` only (across all environments including production). Once merged, the cookbook version bump MR in [`chef-repo`](https://gitlab.com/gitlab-com/gl-infra/chef-repo) is auto-generated - do not merge the follow up chef-repo MR until during the CR. - [x] Prepare [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7338) in [`chef-repo`](https://gitlab.com/gitlab-com/gl-infra/chef-repo) to disable server persistence in production for the main flavour of HAProxy - [x] Prepare [canary MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7339) to update the `canary_api` backend IP and healthcheck - updates `canary_api` role JSON to set `ssl_verify: "no-ssl"` explicitly (was `""`), retain `check_opts: "no-check-ssl port 8181"`, and point the backend at `10.216.8.13:8181`\] - [x] Confirm the K8s `gitlab-webservice-api` Service in GPRD has ready Endpoints on `10.216.8.13:8181` and the IP/port hasn't drifted since the previous CR - [x] Create PAT for gitlab.com with api access > [!note] > > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are listed above. - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [x] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [x] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) **Explicitly call out the persistence-disabled window and MAINT/DRAIN reset risk.** - [x] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - 180 mins > [!note] > > GPRD has \~60 `haproxy-main*` nodes. Cinc server's default max concurrency is 10; we bump to `-C 20` for steps we are confident in, and use a single test node (`haproxy-main-01-lb-gprd`) to validate the rollout before proceeding. This retry adds an explicit HAProxy log check on the test node after the canary IP switch, before the fleet rollout — see Phase 2. #### Phase 1 — Disable server state persistence - [x] Set label ~"change::in-progress" `/label ~change::in-progress` - [x] Merge [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7338) to disable server state persistence for HAProxy main nodes - [x] **Test node:** Run chef-client on a single node first to validate the change ``` knife ssh 'name:haproxy-main-01-lb-gprd.c.gitlab-production.internal AND chef_environment:gprd' 'sudo chef-client' ``` - [x] Validate server state persistence is disabled on the test node ``` knife ssh 'name:haproxy-main-01-lb-gprd.c.gitlab-production.internal AND chef_environment:gprd' \ "sudo grep -E 'server-state-file|load-server-state-from-file' /etc/haproxy/haproxy.cfg | wc -l | awk '\$1==2{print \"yes, present\"} \$1!=2{print \"no, not present\"}'" ``` **We want the test node to respond with "no, not present"** - [x] **Rollout:** Once the test node is validated, roll out to the remaining nodes concurrently (`-C 20`) ``` knife ssh -C 20 'name:haproxy-main* AND chef_environment:gprd' 'sudo chef-client' ``` - [x] Validate server state persistence is disabled across the whole fleet ``` knife ssh 'name:haproxy-main* AND chef_environment:gprd' \ "sudo grep -E 'server-state-file|load-server-state-from-file' /etc/haproxy/haproxy.cfg | wc -l | awk '\$1==2{print \"yes, present\"} \$1!=2{print \"no, not present\"}'" ``` **We want all the nodes to respond with "no, not present"** #### Phase 2 — Switch canary_api backend IPs (route traffic away from NGINX) > [!warning] > > Persistence is now disabled. Minimise the time spent in Phase 2. Any HAProxy reload here will reset MAINT/DRAIN state on all nodes. - [x] Disable chef-client on the HAProxy main nodes ``` knife ssh 'name:haproxy-main* AND chef_environment:gprd' 'sudo /usr/local/bin/chef-client-disable "Re-routing canary traffic away from NGINX (retry) https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22055"' ``` - [x] Merge the auto-generated chef-repo MR that bumps the gitlab-haproxy cookbook version - [x] Merge [canary MR ](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7339)updating the `canary_api` backend IP, healthcheck, and `ssl_verify: "no-ssl"` - [x] Enable chef-client on the test node ``` knife ssh 'name:haproxy-main-01-lb-gprd.c.gitlab-production.internal' 'sudo /usr/local/bin/chef-client-enable' ``` - [x] Run chef-client manually on the test node, and only if chef-client succeeds reload HAProxy ``` knife ssh 'name:haproxy-main-01-lb-gprd.c.gitlab-production.internal' 'sudo chef-client && sudo systemctl reload haproxy' ``` - [x] **Verify rendered config on the test node**: confirm `default-server no-ssl` is present in the `canary_api` backend block and the server line for `gke-cny-api` does **not** carry `ssl verify none` . Also verify, there is no `check-ssl `present ``` knife ssh 'name:haproxy-main-01-lb-gprd.c.gitlab-production.internal' \ "sudo grep -A 20 '^backend canary_api' /etc/haproxy/haproxy.cfg | grep -E 'default-server|server gke-cny-api'" ``` - [x] SSH into the test HAProxy VM and ``` ssh haproxy-main-01-lb-gprd.c.gitlab-production.internal ``` - [x] Verify if the backend server is up ``` echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock \ | awk -F, ' NR==1 { next } $2=="BACKEND" { printf "%-25s %-6s %-12s %-5s %-30s %-18s\n", $1, $18, $37, $38, $57, $74 }' ``` - [x] Test connectivity to the canary API backend using the canary cookie. Using `--connect-to` ensures the request goes through this specific test node's HAProxy (via the internal port `11443`) rather than being load-balanced across the fleet by the GLB. ``` curl -vL \ -H "PRIVATE-TOKEN: ${PAT}" \ -b "gitlab_canary=true" \ --connect-to gitlab.com:443:127.0.0.1:11443 \ "https://gitlab.com/api/v4/user" ``` - [x] **New for retry:** On the same test node, inspect HAProxy logs for live `canary_api` traffic and confirm responses are `200`s targeting `10.216.8.13:8181`, **not** `503 SC-- retries=3 tc=-1`: ``` sudo tail -n 20 /var/log/haproxy.log | grep canary_api | tail -3 ``` ``` echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock | grep gke-cny-api ``` - [x] Look for entries showing `200` against `10.216.8.13:8181` * Any `503 SC--` against `10.216.8.13:8181` is a signal to abort the change and revert the changes (as this change is only rolled out to 1 node at this moment in time, there is an opportunity to investigate with cautious to not roll out to rest of the nodes). - [x] **New for retry:** Cross-check the canary_api backend on the [HAProxy dashboard](https://dashboards.gitlab.net/d/haproxy/haproxy?from=now-30m&orgId=1&refresh=5m&timezone=utc&to=now&var-backend=canary_api&var-datasource=mimir-gitlab-gprd&var-env=gprd) — the test node should appear UP with no error rate spike - [x] Enable chef-client and roll out the canary backend IP change to the remaining HAProxy main nodes concurrently (`-C 20`) ``` knife ssh -C 20 'name:haproxy-main* AND chef_environment:gprd' 'sudo /usr/local/bin/chef-client-enable && sudo chef-client && sudo systemctl reload haproxy' ``` - [ ] **New for retry:** Spot-check HAProxy logs on a sample of post-rollout nodes (e.g. 3–5 nodes) to confirm real traffic is succeeding to `10.216.8.13:8181`: ``` knife ssh 'name:haproxy-main-{02,10,20,30}-lb-gprd.c.gitlab-production.internal' \ "sudo tail -n 20 /var/log/haproxy.log | grep canary_api | tail -3" ``` - [x] Check `canary_api` backend health and error rates in [Grafana](https://dashboards.gitlab.net/d/haproxy/haproxy?from=now-3h&orgId=1&refresh=5m&timezone=utc&to=now&var-backend=canary_api&var-code=$__all&var-datasource=mimir-gitlab-gprd&var-env=gprd&var-frontend=$__all&var-host=$__all&var-interval=$__auto&var-port=&var-server=$__all&viewPanel=panel-27) - [x] Confirm canary NGINX traffic in GPRD is zero in [Grafana](https://dashboards.gitlab.net/d/nginx-main/nginx3a-overview?orgId=1&from=now-3h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=cny&viewPanel=panel-2597844128) — main `api` traffic should remain on NGINX (this is expected and will be migrated in the follow-up CR) #### Phase 3 — Re-enable server state persistence - [x] Revert [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7338) to re-enable server state persistence: \[NEW MR\] - [x] Run chef-client concurrently to roll out the re-enable across all nodes ``` knife ssh -C 20 'name:haproxy-main* AND chef_environment:gprd' 'sudo chef-client' ``` - [x] Confirm persistence is re-enabled on all nodes ``` knife ssh 'name:haproxy-main* AND chef_environment:gprd' \ "sudo grep -E 'server-state-file|load-server-state-from-file' /etc/haproxy/haproxy.cfg | wc -l | awk '\$1==2{print \"yes, present\"} \$1!=2{print \"no, not present\"}'" ``` **We want all the nodes to respond with "yes, present"** - [x] Set label ~"change::complete" `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 60 minutes - [ ] Ensure server persistence is disabled and chef-client is enabled - [ ] Revert the relevant MR(s) (backend IP change and/or cookbook version bump in phase 2): - [ ] canary: revert https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7339 (canary backend) - [ ] revert the cookbook version bump MR (chef-repo) - [ ] Reload HAProxy on the test node and verify `canary-api` service is reachable and backend is healthy - [ ] Roll out the revert to the remaining nodes with ``` knife ssh -C 20 'name:haproxy-main* AND chef_environment:gprd' 'sudo chef-client' ``` - [ ] Revert [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7338) to re-enable server state persistence and run chef-client with `-C 20` - [ ] Roll forward to a new version with reverted changes from [MR](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/460) (version would be an increment rather than a rollback) to add retry config in gitlab-haproxy cookbook - [ ] Merge the auto-generated MR in chef-repo to pin to the new stable version - [ ] Set label ~"change::aborted" `/label ~change::aborted` ## Monitoring ### Key metrics to observe - Metric: HAProxy Dashboard - Location: [HAProxy Dashboard](https://dashboards.gitlab.net/d/haproxy/haproxy?from=now-3h&orgId=1&refresh=5m&timezone=utc&to=now&var-backend=canary_api&var-code=$__all&var-datasource=mimir-gitlab-gprd&var-env=gprd&var-frontend=$__all&var-host=$__all&var-interval=$__auto&var-port=&var-server=$__all&viewPanel=panel-27) - What changes to this metric should prompt a rollback: A sudden significant increase in errors or latency on the `canary_api` backend - Metric: API Dashboard * Location: [API Dashboard](https://dashboards.gitlab.net/d/api-main/api3a-overview?from=now-6h%2Fm&to=now-1m%2Fm&var-environment=gprd&var-stage=cny&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd) * What changes to this metric should prompt a rollback: A sudden significant increase in errors on `canary_api` backend - Metric: API Dashboard - Load Balancer SLI error ratio * Location: [API Dashboard Load Balancer SLI error ratio](https://dashboards.gitlab.net/d/api-main/api3a-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=cny&viewPanel=panel-3255520955) * What changes to this metric should prompt a rollback: A sudden significant increase in errors % as it happened in the last CR