<!-- Please review https://about.gitlab.com/handbook/engineering/infrastructure/change-management/ for the most recent information on our change plans and execution policies. --> # Production Change ### Change Summary The node subnets of the zonal GKE clusters in `gitlab-production` are [slowly approaching saturation](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2077) and will likely start hitting 100% saturation during high traffic hours in the coming 6 months, resulting in not being able to scale to meet traffic demands anymore. The node subnet cannot be replaced without replacing the cluster entirely. The new subnet will have a IP range of size /22 (from /24 currently), allowing up to ~1k nodes, and dropping the subnet saturation to under 25%. We also want to migrate all our clusters to Dataplane V2, which [has a number of advantages](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2#advantages_of) over Calico, and addresses some issues we have encountered in the past: * production-engineering#16775+ * https://gitlab.com/gitlab-com/gl-infra/observability/team/-/issues/3120+ * https://gitlab.com/gitlab-com/gl-infra/observability/team/-/issues/1985+ We have been testing Dataplane V2 in `gstg` for over 2 years since [the unplanned rebuild of the `gstg` zonal GKE clusters](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16909) without any issues. Dataplane V2 also cannot be enabled without replacing the cluster entirely, so this rebuild to replace the node subnet is an opportunity to enable it at the same time. We will rebuild the zonal clusters one at a time, this CR covers the rebuild of the third one: `gprd-us-east1-d`. > [!note] > Auto-deploys will **NOT** need to be blocked during this operation. ### Change Details <!-- To automatically add your change to the GitLab Production calendar update the following fields: - Time tracking - Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) Bot: https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler --> 1. **Services Impacted** - ~"Service::Kube" 1. **Change Technician** - `@pguinoiseau` `@ayeung` 1. **Change Reviewer** - @jcstephenson 1. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2025-07-31 03:00 1. **Time tracking** - 4 hours 1. **Downtime Component** - `gprd-us-east1-d` GKE cluster (GitLab.com availability unaffected) ### Set Maintenance Mode in GitLab If your change involves scheduled maintenance, add a step to set and [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Detailed steps for the change ### Pre-execution steps - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [x] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In `#production` channel, mention `@sre-oncall` and this issue and await their acknowledgement.) - [x] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [x] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change *Estimated Time to Complete (mins)* - 240 - [x] Set label ~"change::in-progress" `/label ~change::in-progress` - [x] Set the variable `CLUSTER_SKIP` to `gprd-us-east1-d` in the [`ops` mirror CI variables](https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/settings/ci_cd) - [x] Silence all alerts for the cluster: - [x] [Cluster alerts](https://alerts.gitlab.net/#/silences/new?filter=%7Bcluster%3D%22gprd-us-east1-d%22%2C%20env%3D%22gprd%22%7D) - [x] [Traffic cessation](https://alerts.gitlab.net/#/silences/new?filter=%7Balert_class%3D%22traffic_cessation%22%2C%20env%3D%22gprd%22%2C%20region%3D%22us-east1-d%22%7D) - [x] Gracefully stop all HAProxy nodes in the `gprd-us-east1-d` zone 5 at a time, 30 seconds apart: ```sh knife ssh -C 5 'chef_environment:gprd AND roles:gprd-base-haproxy AND zone:projects\/805818759045\/zones\/us-east1-d' \ 'sudo systemctl mask haproxy.service; sudo systemctl kill --signal SIGUSR1 haproxy.service; while [ $(systemctl is-active haproxy.service) != "inactive" ] && [ ${i:=1} -lt 150 ]; do sleep 2; i=$((i + 1)); done; sudo systemctl stop haproxy.service; systemctl status haproxy.service; sleep 30' ``` > [!note] > `SIGUSR1` signals HAProxy to stop listening for new connections and let all current connections finish normally before exiting. The default stop signal in `haproxy.service` is SIGTERM which closes all connections immediately, so we can't simply use `systemctl stop haproxy.service` here. - [x] Check in the [HAProxy dashboard](https://dashboards.gitlab.net/d/haproxy/haproxy?var-interval=$__auto&orgId=1&from=now-1h&to=now&timezone=utc&var-datasource=mimir-gitlab-gprd&var-env=gprd&var-host=$__all&var-port=&var-backend=$__all&var-frontend=$__all&var-server=pages-us-east1-d&var-server=ws-gke-us-east1-d&var-server=api-gke-us-east1-d&var-server=web-gke-us-east1-d&var-server=registry-us-east1-d&var-server=shell-gke-us-east1-d&var-server=git-https-gke-us-east1-d&var-server=ai-assisted-gke-us-east1-d&var-code=$__all&refresh=5m) that all traffic to the `us-east1-d` backend servers has ceased. - [x] Disable deletion protection for the cluster: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/11603 - [x] Recreate the subnet and GKE cluster in https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/11548: - [x] Perform a `terraform plan` via Atlantis in `gprd` by commenting in the MR: ```sh atlantis plan -p gprd -- -replace module.gke-us-east1-d.google_container_cluster.cluster ``` It should show something like: ``` Terraform will perform the following actions: # module.gke-us-east1-d.google_container_cluster.cluster will be replaced, as requested ... # module.gke-us-east1-d.google_container_node_pool.node_pool["generic-1"] will be replaced due to changes in replace_triggered_by ``` - [x] Then apply **without automerging** by commenting in the MR: ```sh atlantis apply -p gprd --auto-merge-disabled ``` > [!note] > This should take around 20 to 30 minutes. Once applied, the new cluster and all its node pools should be up. - [x] In the [GCP console](https://console.cloud.google.com/kubernetes/clusters/details/us-east1-d/gprd-us-east1-d/nodes?inv=1&invt=Ab4MLA&project=gitlab-production), manually resize `generic-3` and `generic-4` node pools to 60 nodes each. - [x] Perform a `terraform plan` via Atlantis in the `vault-production` environment to update the Kubernetes authentication method and secret engine by commenting in the MR: ```sh atlantis plan -p vault-production ``` - [x] Then `apply` by commenting in the MR: ```sh atlantis apply -p vault-production ``` Once applied, Atlantis will merge the MR automatically. - [x] Take note of the new reserved IPs for GitLab services: ```sh gcloud compute addresses list --project gitlab-production | grep -E '(ai-assisted|api|git-https|nginx|pages|registry|ssh|web|websockets)-gke-gprd-us-east1-d' ``` - [x] In [`chef-repo`](https://gitlab.com/gitlab-com/gl-infra/chef-repo), open a MR replacing the backend server IPs for the zone `us-east1-d` with the new IPs collected above, get it approved and merge it: ```yaml # roles/gprd-base-haproxy-main-config.json "api-gke-us-east1-d": "<nginx-gke-gprd-us-east1-d>" "web-gke-us-east1-d": "<web-gke-gprd-us-east1-d>" "shell-gke-us-east1-d": "<ssh-gke-gprd-us-east1-d>" "ws-gke-us-east1-d": "<websockets-gke-gprd-us-east1-d>" "ai-assisted-gke-us-east1-d": "<ai-assisted-gke-gprd-us-east1-d>" "git-https-gke-us-east1-d": "<git-https-gke-gprd-us-east1-d>" # roles/gprd-base-haproxy-registry.json "registry-us-east1-d": "<registry-gke-gprd-us-east1-d>" # roles/gprd-base-haproxy-pages.json "pages-us-east1-d": "<pages-gke-gprd-us-east1-d>" ``` - [x] Update the service LB IPs for the zone `us-east1-d` with the new IPs collected above in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/4617 **but don't merge it yet**. - [x] Run `chef-client` on all HAProxy nodes ```sh knife ssh -C 10 'chef_environment:gprd AND roles:gprd-base-haproxy' 'sudo chef-client' ``` > [!note] > This can continue in the background while doing the next steps up until ready to re-enable the HAProxy backend servers > [!note] > Letting `chef-client` run on its own for the next 40 minutes is also an option if not running against the clock - [x] Get the new cluster credentials and verify that the cluster can be accessed with `kubectl`: ```sh gcloud container clusters get-credentials gprd-us-east1-d --project gitlab-production --location us-east1-d # or glsh kube setup glsh kube use-cluster gprd-us-east1-d kubectl get pods --all-namespaces ``` - [x] From the `gitlab-helmfiles` repository, pull the latest changes and install the `vault-k8s-secrets` release: ```sh git pull cd releases/vault-k8s-secrets helmfile -e gprd-us-east1-d apply ``` - [x] Get the new JWT token that was just provisioned with the release and save it into Vault: ```sh kubectl --namespace vault-k8s-secrets get secret vault-k8s-secrets-token -o jsonpath='{.data.token}' | base64 -d | \ vault kv put ci/ops-gitlab-net/gitlab-com/gl-infra/config-mgmt/vault-production/kubernetes/clusters/gprd/gprd-us-east1-d service_account_jwt=- ``` - [x] [Trigger a new `config-mgmt` pipeline for the `vault-production` environment](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/pipelines/new?ref=main&var[ENV]=vault-production) to update the Kubernetes secrets engine with this new JWT token. - [x] Add the necessary annotations and labels to the `kube-dns` configmap so that `gitlab-helmfiles` can manage it via Helm: ```sh kubectl -n kube-system annotate configmap/kube-dns meta.helm.sh/release-name=kube-dns-extras meta.helm.sh/release-namespace=kube-system kubectl -n kube-system label configmap/kube-dns app.kubernetes.io/managed-by=Helm ``` - [x] From [`gitlab-helmfiles` CI pipelines](https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/pipelines), find the latest default branch pipeline, and re-run the apply job for `gprd-us-east1-d`. - [x] From [`tanka-deployments` CI pipelines](https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/tanka-deployments/-/pipelines), find the latest default branch pipeline, and re-run the apply jobs for `gprd-us-east1-d`. - [x] Run `kubectl get pods --all-namespaces` and check that all workloads are working correctly. - [x] Remove the variable `CLUSTER_SKIP` from the [`ops` mirror CI variables](https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/settings/ci_cd) - [x] Merge https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/4617 to deploy the GitLab releases. > [!important] > This will install all releases and configurations but will not deploy the > correct version of GitLab, that comes in the following step. - [x] Deploy the correct version of GitLab by re-triggering the Kubernetes pipeline of the latest successful `auto-deploy` job: - [x] Go to the [`#announcements` channel](https://gitlab.slack.com/archives/C8PKBH3M5) and check the latest successful job for the targeted environment. - [x] Retry the `kubernetes-gprd` trigger job. - [x] Spot check the cluster to validate that all pods are coming online and remain in a running state: ```sh glsh kube use-cluster gprd-us-east1-d kubectl get pods --namespace gitlab ``` - [x] Verify that we run the same version of GitLab on all clusters: ```sh glsh kube use-cluster gprd-us-east1-c kubectl get configmap --namespace gitlab gitlab-gitlab-chart-info -o jsonpath="{.data.gitlabVersion}" glsh kube use-cluster gprd-us-east1-d kubectl get configmap --namespace gitlab gitlab-gitlab-chart-info -o jsonpath="{.data.gitlabVersion}" ``` The version from both clusters should match. - [x] Check that [this dashboard](https://dashboards.gitlab.net/goto/Z3RsDWwNg?orgId=1) is showing the numbers pods and containers for this cluster. - [x] Remove any silences that were created earlier. - [x] Validate that no alerts are firing related to this cluster in [Alertmanager](https://alerts.gitlab.net). - [x] Restart all HAProxy nodes in the `gprd-us-east1-d` zone one at a time, 30 seconds apart: ```sh knife ssh -C 1 'chef_environment:gprd AND roles:gprd-base-haproxy AND zone:projects\/805818759045\/zones\/us-east1-d' \ 'sudo systemctl unmask haproxy.service; sudo systemctl start haproxy.service; systemctl status haproxy.service; sleep 30' ``` - [x] Check in the [HAProxy dashboard](https://dashboards.gitlab.net/d/haproxy/haproxy?var-interval=$__auto&orgId=1&from=now-1h&to=now&timezone=utc&var-datasource=mimir-gitlab-gprd&var-env=gprd&var-host=$__all&var-port=&var-backend=$__all&var-frontend=$__all&var-server=pages-us-east1-d&var-server=ws-gke-us-east1-d&var-server=api-gke-us-east1-d&var-server=web-gke-us-east1-d&var-server=registry-us-east1-d&var-server=shell-gke-us-east1-d&var-server=git-https-gke-us-east1-d&var-server=ai-assisted-gke-us-east1-d&var-code=$__all&refresh=5m) that all traffic to the `us-east1-d` backend servers has resumed. - [x] Party! :partyparrot: - [x] Set label ~"change::complete" `/label ~change::complete` ## Rollback > [!important] > There is no rollback possible once the GKE cluster has been destroyed ### Rollback steps - steps to be taken in the event of a need to rollback this change *Estimated Time to Complete (mins)* - 10 - [ ] Remove any silences that were created earlier. - [ ] Restart all HAProxy nodes in the `gprd-us-east1-d` zone one at a time, 30 seconds apart: ```sh knife ssh -C 1 'chef_environment:gprd AND roles:gprd-base-haproxy AND zone:projects\/805818759045\/zones\/us-east1-d' \ 'sudo systemctl unmask haproxy.service; sudo systemctl start haproxy.service; systemctl status haproxy.service; sleep 30' ``` - [ ] Set label ~"change::aborted" `/label ~change::aborted` ## Monitoring ### Key metrics to observe <!-- * Describe which dashboards and which specific metrics we should be monitoring related to this change using the format below. --> - Metric: Everything - Location: https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage - What changes to this metric should prompt a rollback: any significant apdex drop or error increase after putting the `us-east1-d` backend servers in `MAINT` mode; nothing after destroying the GKE cluster, the only path is forward from that point. - Metric: HAProxy `us-east1-d` backend servers - Location: https://dashboards.gitlab.net/d/haproxy/haproxy?var-interval=$__auto&orgId=1&from=now-1h&to=now&timezone=utc&var-datasource=mimir-gitlab-gprd&var-env=gprd&var-host=$__all&var-port=&var-backend=$__all&var-frontend=$__all&var-server=pages-us-east1-d&var-server=ws-gke-us-east1-d&var-server=api-gke-us-east1-d&var-server=web-gke-us-east1-d&var-server=registry-us-east1-d&var-server=shell-gke-us-east1-d&var-server=git-https-gke-us-east1-d&var-server=ai-assisted-gke-us-east1-d&var-code=$__all&refresh=5m ## Change Reviewer checklist <!-- To be filled out by the reviewer. --> ~C4 ~C3 ~C2 ~C1: - [ ] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ## Change Technician checklist <!-- To find out who is on-call, use the `@sre-oncall` handle in slack --> - [ ] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [ ] This Change Issue is linked to the appropriate Issue and/or Epic - [ ] Change has been tested in staging and results noted in a comment on this issue. - [ ] A dry-run has been conducted and results noted in a comment on this issue. - [ ] The change execution window respects the [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl). - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar. - [ ] For ~C1 and ~C2 change issues, the Infrastructure Manager provided approval with the ~manager_approved label on the issue. Mention `@gitlab-org/saas-platforms/inframanagers` in this issue to request approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.)