# Production Change ## Change Summary This change redirects production main api traffic to directly reach the API backends in Kubernetes without going through the NGINX controller from HAProxy. ## Change Details 1. **Services Impacted** - ~&quot;Service::API&quot; ~&quot;Service::NGINX&quot; 2. **Change Technician** - @haseeb_ahmed @thisisshreya 3. **Change Reviewer** - @cmcfarland / @pguinoiseau 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-06-05 10:00 5. **Time tracking** - 180 minutes 6. **Downtime Component** - none > [!important] > > If your change involves scheduled maintenance, add a step to set and [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) per our runbooks. This will make sure SLA calculations adjust for the maintenance period. > [!warning] > > **Server state persistence window:** During this change, HAProxy server state persistence will be temporarily disabled. While persistence is disabled, any HAProxy reload/restart will reset MAINT/DRAIN state — meaning any operator-set drains will silently revert. This will be communicated to release managers as part of this C1 CR, and **the window between disabling and re-enabling persistence must be kept as short as possible**. See [context](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28303#note_3128214144). ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~&quot;change::scheduled&quot; ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~&quot;blocks deployments&quot; and/or ~&quot;blocks feature-flags&quot; are applied as necessary. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl). - [ ] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [ ] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~&quot;blocks deployments&quot; change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps - [x] <span dir="">Prepare and merge </span>[<span dir="">MR</span>](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/456)<span dir=""> in </span>[<span dir="">`gitlab-haproxy`</span>](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy)<span dir=""> to extend the retry/timeout parity config to production. Once merged, the cookbook version bump MR in </span>[`chef-repo`](https://gitlab.com/gitlab-com/gl-infra/chef-repo)<span dir=""> is auto-generated - do not merge the follow up chef-repo MR until during the CR.</span> - [ ] Prepare [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7392) in [`chef-repo`](https://gitlab.com/gitlab-com/gl-infra/chef-repo) to disable server persistence in production for the main flavour of HAProxy - [ ] Prepare [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7225) to update `main API`backend IP and healthcheck - [ ] Create PAT for gitlab.com with api access > [!note] > > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are listed above. - [ ] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [ ] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [ ] For ~C1, ~C2, or ~&quot;blocks deployments&quot; change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) **Explicitly call out the persistence-disabled window and MAINT/DRAIN reset risk.** - [ ] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - 180 mins > [!note] > > GPRD has \~60 `haproxy-main*` nodes. Cinc server's default max concurrency is 10; we bump to `-C 20` for steps we are confident in, and use a single test node (`haproxy-main-07-lb-gprd`) to validate the rollout before proceeding. This retry adds an explicit HAProxy log check on the test node after the IP switch, before the fleet rollout — see Phase 2. #### Phase 1 — Disable server state persistence - [x] Set label ~&quot;change::in-progress&quot; `/label ~change::in-progress` - [x] Merge [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7392) to disable server state persistence for HAProxy main nodes - [x] **Rollout:** Roll the changes out to all the nodes concurrently (`-C 20`) ``` knife ssh -C 20 'name:haproxy-main* AND chef_environment:gprd' 'sudo chef-client' ``` - [x] Validate server state persistence is disabled across the whole fleet ( ps: it takes a few minutes for the server state persistence to be disabled across all the nodes ) ``` knife ssh 'name:haproxy-main* AND chef_environment:gprd' \ "sudo grep -E 'server-state-file|load-server-state-from-file' /etc/haproxy/haproxy.cfg | wc -l | awk '\$1==2{print \"yes, present\"} \$1!=2{print \"no, not present\"}'" ``` **We want all the nodes to respond with "no, not present"** #### Phase 2 — Switch API backend IPs (route traffic away from NGINX) > [!warning] > > Persistence is now disabled. Minimise the time spent in Phase 2. Any HAProxy reload here will reset MAINT/DRAIN state on all nodes. - [x] Disable chef-client on the HAProxy main nodes ``` knife ssh 'name:haproxy-main* AND chef_environment:gprd' 'sudo /usr/local/bin/chef-client-disable "Re-routing main api traffic away from NGINX (retry) https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22152"' ``` - [x] Merge the [auto-generated chef-repo MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7472) that bumps the gitlab-haproxy cookbook version in Production - [x] Merge [MR ](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7225)updating the `API` backend IP, healthcheck ##### Test node rollout & validation - [x] **new for retry** :Drain the test node to ensure no new connections are made and all existing connections terminate sucessfully ``` sudo /usr/local/sbin/drain_haproxy.sh ``` - [x] Enable chef-client on the test node ``` knife ssh 'name:haproxy-main-07-lb-gprd.c.gitlab-production.internal' 'sudo /usr/local/bin/chef-client-enable' ``` - [x] Run chef-client manually on the test node, and only if chef-client succeeds reload HAProxy ``` knife ssh 'name:haproxy-main-07-lb-gprd.c.gitlab-production.internal' 'sudo chef-client && sudo systemctl reload haproxy' ``` - [x] **Verify rendered config on the test node**: confirm `default-server no-ssl` is present in the `api` backend block and the server line for `gke-cny-api` does **not** carry `ssl verify none` . Also verify, there is no `check-ssl `present ``` knife ssh 'name:haproxy-main-07-lb-gprd.c.gitlab-production.internal' \ "sudo grep -A 20 '^backend api' /etc/haproxy/haproxy.cfg | grep -E 'default-server|server gke-cny-api'" ``` - [x] SSH into the test HAProxy VM and ``` ssh haproxy-main-07-lb-gprd.c.gitlab-production.internal ``` - [x] Verify if the backend server is up ``` echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock \ | awk -F, ' NR==1 { next } $2=="BACKEND" { printf "%-25s %-6s %-12s %-5s %-30s %-18s\n", $1, $18, $37, $38, $57, $74 }' ``` - [x] Test connectivity to the API backend. Using `--connect-to` ensures the request goes through this specific test node's HAProxy (via the internal port `11443`) rather than being load-balanced across the fleet by the GLB. ``` curl -vL \ -H "PRIVATE-TOKEN: ${PAT}" \ --connect-to gitlab.com:443:127.0.0.1:11443 \ "https://gitlab.com/api/v4/user" ``` - [x] Undrain the node so it can start accepting live traffic ``` sudo /usr/local/sbin/drain_haproxy.sh -u ``` - [x] On the same test node, inspect HAProxy logs for main API traffic and confirm responses are `200`s ``` sudo tail -n 20 /var/log/haproxy.log | grep api | tail -3 ``` ``` echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock | grep gke-cny-api ``` - [x] Look for entries showing `200` * Any `503 SC--` is a signal to abort the change and revert the changes (as this change is only rolled out to 1 node at this moment in time, there is an opportunity to investigate with cautious to not roll out to rest of the nodes). - [x] Cross-check the api backend on the [HAProxy dashboard](https://dashboards.gitlab.net/d/haproxy/haproxy?from=now-30m&orgId=1&refresh=5m&timezone=utc&to=now&var-backend=canary_api&var-datasource=mimir-gitlab-gprd&var-env=gprd) — the test node should appear UP with no error rate spike ##### Batched Fleet-wide rollout & validation - [ ] Enable chef-client and roll out the backend IP change to the first 10 nodes ( except the test node ) ``` knife search node 'name:haproxy-main* AND chef_environment:gprd' -i \ | sort \ | tail -n +2 \ | head -10 \ | while read -r node; do echo "=== $node ===" knife ssh "name:$node" \ 'sudo /usr/local/bin/chef-client-enable && sudo chef-client && sudo systemctl reload haproxy' done ``` - [ ] Check HAProxy logs on a sample of rolled out nodes to confirm real traffic is succeeding: ``` knife ssh 'name:haproxy-main-02-lb-gprd.c.gitlab-production.internal' \ "sudo tail -n 20 /var/log/haproxy.log | grep api | tail -3" ``` - [ ] Look for entries showing `200` * Any `503 SC--` is a signal to abort the change and revert the changes (as this change is only rolled out to 1 node at this moment in time, there is an opportunity to investigate with cautious to not roll out to rest of the nodes). - [ ] Cross-check the api backend on the [HAProxy dashboard](https://dashboards.gitlab.net/d/haproxy/haproxy?from=now-30m&orgId=1&refresh=5m&timezone=utc&to=now&var-backend=canary_api&var-datasource=mimir-gitlab-gprd&var-env=gprd) — the nodes should appear UP with no error rate spike - [ ] Enable chef-client and roll out the backend IP change to the remaining HAProxy main nodes concurrently (`-C 20`) ``` knife search node 'name:haproxy-main* AND chef_environment:gprd' -i \ | sort \ | sed -n '11,60p' \ | while read -r node; do echo "=== $node ===" knife ssh -C 20 "name:$node" \ 'sudo /usr/local/bin/chef-client-enable && sudo chef-client && sudo systemctl reload haproxy' done ``` - [ ] Spot-check HAProxy logs on a sample of post-rollout nodes (e.g. 3–5 nodes) to confirm real traffic is succeeding: ``` knife ssh 'name:haproxy-main-{02,10,20,30}-lb-gprd.c.gitlab-production.internal' \ "sudo tail -n 20 /var/log/haproxy.log | grep api | tail -3" ``` - [ ] Check `api` backend health and error rates in [Grafana](https://dashboards.gitlab.net/d/haproxy/haproxy?from=now-30m&orgId=1&refresh=5m&timezone=utc&to=now&var-backend=api&var-datasource=mimir-gitlab-gprd&var-env=gprd&var-interval=$__auto&var-host=$__all&var-port=&var-frontend=$__all&var-server=$__all&var-code=$__all) - [ ] Confirm NGINX traffic in GPRD is zero in [Grafana](https://dashboards.gitlab.net/d/nginx-main/nginx3a-overview?orgId=1&from=now-3h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&viewPanel=panel-2597844128) — main `api` traffic should remain on NGINX (this is expected and will be migrated in the follow-up CR) #### Phase 3 — Re-enable server state persistence - [ ] Revert [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7392) to re-enable server state persistence: - [ ] Run chef-client concurrently to roll out the re-enable across all nodes ``` knife ssh -C 20 'name:haproxy-main* AND chef_environment:gprd' 'sudo chef-client' ``` - [ ] Confirm persistence is re-enabled on all nodes ``` knife ssh 'name:haproxy-main* AND chef_environment:gprd' \ "sudo grep -E 'server-state-file|load-server-state-from-file' /etc/haproxy/haproxy.cfg | wc -l | awk '\$1==2{print \"yes, present\"} \$1!=2{print \"no, not present\"}'" ``` **We want all the nodes to respond with "yes, present"** - [ ] Set label ~&quot;change::complete&quot; `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 60 minutes - [x] Ensure server persistence is disabled and chef-client is enabled - [x] Revert the relevant MR(s) (backend IP change and/or cookbook version bump in phase 2): - [ ] main API: revert https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7225 (main api backend) - [ ] revert the cookbook version bump MR (chef-repo) - [x] Reload HAProxy on the test node and verify `main api` service is reachable and backend is healthy - [x] Roll out the revert to the remaining nodes with ``` knife ssh -C 20 'name:haproxy-main* AND chef_environment:gprd' 'sudo chef-client' ``` - [x] Revert [MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7392) to re-enable server state persistence and run chef-client with `-C 20` - [ ] Roll forward to a new version with reverted changes from [MR](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/456) (version would be an increment rather than a rollback) to add retry config in gitlab-haproxy cookbook - [ ] Merge the auto-generated MR in chef-repo to pin to the new stable version - [ ] Set label ~&quot;change::aborted&quot; `/label ~change::aborted` ## Monitoring ### Key metrics to observe - Metric: HAProxy Dashboard - Location: [HAProxy Dashboard](https://dashboards.gitlab.net/d/haproxy/haproxy?from=now-3h&orgId=1&refresh=5m&timezone=utc&to=now&var-backend=api&var-code=$__all&var-datasource=mimir-gitlab-gprd&var-env=gprd&var-frontend=$__all&var-host=$__all&var-interval=$__auto&var-port=&var-server=$__all&viewPanel=panel-27) - What changes to this metric should prompt a rollback: A sudden significant increase in errors or latency on the `main api` backend - Metric: API Dashboard * Location: [API Dashboard](https://dashboards.gitlab.net/d/api-main/api3a-overview?from=now-6h%2Fm&to=now-1m%2Fm&var-environment=gprd&var-stage=main&timezone=utc&orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd) * What changes to this metric should prompt a rollback: A sudden significant increase in errors on `main api` backend - Metric: API Dashboard - Load Balancer SLI error ratio * Location: [API Dashboard Load Balancer SLI error ratio](https://dashboards.gitlab.net/d/api-main/api3a-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&viewPanel=panel-3255520955) * What changes to this metric should prompt a rollback: A sudden significant increase in errors % as it happened in the last CR - Metric: Nginx - RPS * Location: [Nginx RPS](https://dashboards.gitlab.net/d/nginx-main/nginx3a-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&viewPanel=panel-2597844128)