<!--Please review https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/ for the most recent information on our change plans and execution policies.--> # Production Change ## Change Summary Migrate **`ops.gitlab.net`** (the `gitlab-ops` ArgoCD app — the ApplicationSet-generated resource name is `gitlab-ops--1a387`, which is what the `kubectl … application` commands and the ArgoCD UI steps below reference) off the **in-cluster Bitnami Redis subchart** and onto the external **Memorystore Valkey** instance over a PSC endpoint, using **TLS (`rediss`)** (the instance IP is assigned when Terraform creates it in-window). GitLab trusts a custom CA and authenticates with an AUTH token, both synced from Vault via ExternalSecrets. Because the in-cluster Redis holds Sidekiq queues, cache, and sessions, the cutover happens in a **maintenance window** with a Redis snapshot restored into Valkey. ArgoCD autosync for `gitlab-ops` is disabled first (via git) so the cutover is applied **deliberately** after the data is migrated, not automatically on merge. The `rediss` + CA + auth path was validated against a **test** Valkey instance (`PING → PONG` from the toolbox using the exact GitLab Ruby redis stack), and the chart render (`helm template … 9.11.9`) confirms every redis instance renders `rediss://…:6379` with `ssl_params.ca_file`. Note: [config-mgmt MR 15068](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15068) removes that test instance, so **no Valkey exists until the change window creates it**; real connectivity is proven in-window once Sidekiq is running on Valkey. Change criticality: **~C2** (decided) — planned maintenance window on the internal ops instance, with Redis state migration. Issue: https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1671 ## Known constraint: maintenance mode on ops.gitlab.net blocks git writes Discovered during change planning. Maintenance mode on `ops.gitlab.net` blocks **all** git pushes and API writes to that instance (gitlab-shell returns 401 on `/internal/allowed`; JSON `POST`/`PUT`/`PATCH`/`DELETE` return 403). This affects the change plan in two independent places: **1. ArgoCD reads from a mirror on ops.gitlab.net, not from gitlab.com.** The `argocd-apps` Application's `spec.source.repoURL` is `https://ops.gitlab.net/gitlab-com/gl-infra/argocd/apps.git`. The canonical repo is on gitlab.com and is **push-mirrored** to ops.gitlab.net. While maintenance mode is on, that mirror push cannot land, so merging MR 3 on gitlab.com in-window would be invisible to ArgoCD until maintenance mode is lifted, which is a deadlock (Sidekiq must be back on Valkey *before* maintenance mode comes off). > **Mitigation:** MR 3 is not merged in-window. The real Valkey IP and the two new Vault secret versions are applied directly to the `gitlab-ops--1a387` ArgoCD `Application` as Helm parameter overrides. MR 3 is finalized and merged after the window, and the override is then removed. **2. config-mgmt lives on ops.gitlab.net, so [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) cannot be merged or applied via CI in-window.** The Valkey instance must be created *inside* the window (its `gcs_source` restore consumes the quiesced dump), but the MR cannot be marked ready or merged, and the CI apply path is blocked. **config-mgmt applies run through Atlantis, which is comment-driven (`atlantis apply` on the MR); maintenance mode makes `ops.gitlab.net` read-only, so comment creation is blocked too** — even a ready MR could not be applied via CI. > **Mitigation:** apply the Terraform manually from the MR branch during the window. The backend is **GCS** (`gitlab-infra-tf-ops`, see `environments/ops/remote_backend.tf`), not GitLab-managed state, so state read/write/locking are unaffected by maintenance mode. [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) is merged immediately after maintenance mode is lifted. > > **This creates a temporary state/config divergence:** after the manual apply, the Terraform state contains `google_memorystore_instance.gitlab_valkey`, `vault_kv_secret_v2.gitlab_valkey_ca`, and the `gitlab_valkey_psc_auto_connection` output, but `main` still has all three commented out. **Any apply against the ops environment from `main` before [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) merges will plan to destroy them**, and `deletion_protection_enabled = false` (deliberately, to keep re-seed possible) will not stop it. Merging [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) is therefore time-critical, not cleanup. ### Consequence: ExternalSecret names temporarily lag their Vault versions Creating the Valkey instance mints a **new auth token** and a **new CA**, so the Vault versions become `memorystore-valkey/auth` **v2** and `memorystore-valkey/tls` **v3**. By convention the ExternalSecret resources would be renamed to match (`memorystore-valkey-auth-v2`, `memorystore-valkey-tls-v3`), but that rename is a `values-vault-secrets.yaml` change on gitlab.com, which cannot reach ArgoCD in-window (constraint 1 above). > **Mitigation (deliberate, temporary):** keep the existing `memorystore-valkey-auth-v1` / `memorystore-valkey-tls-v1` resource names in-window and override only the pinned `remoteRef.version` to `2` and `3`. For the duration of the window and until the follow-up MR merges, **a Secret named `…-v1` holds v2/v3 content**. This is understood and accepted; the rename is done properly in the post-migration MR. > > The naming convention normally also serves to force a pod rollout (a new Secret name changes the pod spec). That is covered here by the explicit `kubectl rollout restart` in the cutover step, so nothing is lost by deferring the rename. ## Execution notes & post-mortem (executed 2026-08-24) The migration **succeeded** — `ops.gitlab.net` runs fully on Memorystore Valkey over `rediss` (webservice, sidekiq, kas healthy; in-cluster Redis fully decommissioned — StatefulSet, PVC, and backing PD all deleted). But execution hit six issues the plan did not anticipate. They are documented here and folded into the steps below. **1. The target is an ApplicationSet-generated app — patch the ApplicationSet, not the Application.** `gitlab-ops--1a387` is generated by the **`service-gitlab-ops` ApplicationSet** (`applicationsSync: create-update`). Patching the Application's `spec.sources[*].helm.parameters` directly **does not stick** — the ApplicationSet controller reconciles the generated Application back to its template within seconds and strips the override. The override must go on the **ApplicationSet template** (`/spec/template/spec/sources/N/helm/parameters`): `sources[0]` = `gitlab` chart (`global.redis.*`), `sources[2]` = `vault-secrets` chart (`externalSecrets.*`). It persists in-window because `argocd-apps` (the app-of-apps) has `selfHeal: false` and the git mirror is frozen under maintenance mode. **2. Autosync was still ON — MR 2 could never disable it via git.** MR 2 set `automated.enabled: false` in the ApplicationSet **template**, but the ApplicationSet's `ignoreApplicationDifferences` includes `/spec/syncPolicy/automated/enabled`. That ignore rule cuts **both ways**: it lets a human toggle autosync on the generated app, but it also means a template change to that field is **never propagated** to an already-existing Application. So the live app kept `automated.enabled: true`. Consequence: applying the override to the ApplicationSet template (issue 1) changed desired state and **triggered an automatic Sync**, deploying the cutover immediately and bypassing the intended "verify Secrets, then deliberate manual Sync" gate. **To actually disable autosync, patch the Application directly** (`kubectl patch application … -p '{"spec":{"syncPolicy":{"automated":{"enabled":false}}}}'`); the same ignore rule then protects the manual value from being reverted. **3. `refreshInterval: 0` → ESO never updates an existing Secret; you must delete it.** Bumping the pinned `remoteRef.version` (v1→v2/v3) updated the ExternalSecret spec but the underlying Kubernetes **Secret content did not change**. With `refreshInterval: 0`, ESO writes the Secret **once at creation** and never reconciles it again — neither the version change, a `force-sync` annotation, nor an ESO controller restart forced a re-pull. (Evidence: every `refreshInterval: 0` ExternalSecret in the namespace shows `refreshTime` = its creation date, some >1 year old; the only one that had refreshed was `rails-secrets-v2`, because the `-v2` rename made it a brand-new object.) **Fix: delete the target Secret** — ESO (`creationPolicy: Owner`) recreates it via the create path, pulling the pinned version. Safe because ESO recreates within seconds and the Secret is ESO-owned, not ArgoCD-managed. This **validates the runbook's original "rename the ExternalSecret to a new `-vN` object" convention**; the in-place version-bump shortcut does not work. **4. HPA will not scale Sidekiq up from 0.** The quiesce step scaled the Sidekiq Deployment to 0. Recreating the HPA at cutover does **not** bring it back — an HPA refuses to scale a Deployment from 0 (`scaling is disabled since the replica count of the target is zero`), and ArgoCD ignores `/spec/replicas` on HPA-managed Deployments so it won't fix it either. **Fix: manually scale the Deployment to `minReplicas` after the cutover**; the HPA then takes over. **5. Order-of-operations fallout & cosmetic ArgoCD state.** Because autosync auto-deployed before the Secrets were fixed, the first wave of pods (kas, webservice) came up with stale v1 creds and crash-looped/Pended until the Secrets were deleted-and-recreated (issue 3) and the deployments were rolled. After terminating the stuck auto-sync operation, the app shows red (`Sync failed: Operation terminated`, health frozen at an old `Degraded` snapshot) — this is **cosmetic**; zero child resources are unhealthy. A hard refresh clears it. Do **not** Sync to "fix" it (autosync is off deliberately; a Sync would re-run migrations and risk pruning). **6. Pruning the Redis subchart leaves its PVC orphaned — delete it by hand.** Merging the teardown MR (`redis.install: false`) and syncing with prune removed the `ops-gitlab-redis-master` StatefulSet but **not** its PVC (`redis-data-ops-gitlab-redis-master-0`, 8Gi). Deleting a StatefulSet does not cascade to its PVCs by default, and ArgoCD does not prune them because they were dynamically provisioned by the StatefulSet controller, not rendered from Helm/git — so they are not in ArgoCD's desired-state tree. The PVC had to be deleted manually; its PV reclaim policy was `Delete`, so that also destroyed the backing PD and the old Redis data. (The runbook wrongly claimed `prune: true` removes the PVCs.) **Corrected sequence for next time:** (a) verify autosync is actually OFF on the live Application — don't trust a git MR; (b) apply the override to the **ApplicationSet template** sources[0]/[2]; (c) **delete** the two target Secrets so ESO re-pulls v2/v3, confirming `data-hash` changes; (d) roll the GitLab deployments; (e) **manually scale Sidekiq to minReplicas**, confirm HPA `ScalingActive: True`; (f) scale down in-cluster Redis, verify, lift maintenance mode, smoke test; (g) post-soak, merge the teardown MR + sync, then **manually delete the orphaned redis PVC**. ## Related merge requests and issues **This change (argocd/apps, three ordered MRs):** - MR 1: add Memorystore Valkey ExternalSecrets (`values-vault-secrets.yaml`) — **merged**; the Secrets are on `main` and synced in-cluster (`version: "1"` points at the Phase 0 test instance and is overridden in-window): https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3060 - MR 2: disable ArgoCD autosync for `gitlab-ops` (`service.yaml`): https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3059 - MR 3 (Draft): Redis to Valkey TLS cutover — `values.yaml` `global.redis` **repoint only**: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3058 - The in-cluster Redis subchart (`redis.install: true`) and `redis-secret-v1` are deliberately left in place as a warm rollback target; their teardown is a separate post-migration MR. - Finalized at cutover time with the real instance details: `global.redis.host` (IP) and the ExternalSecret `version:`s are only known once Terraform creates the instance and restores the dump. The current `10.253.11.2` / `version: "1"` are placeholders. - **Not merged in-window** — see "Known constraint" above. Applied live via ArgoCD Application Helm parameter overrides instead, then reconciled into git after the window. - MR 4 (Draft, post-migration): tear down the in-cluster Redis (`redis.install: false` + remove `redis-secret-v1`) once Valkey is confirmed good: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3092 - MR 5 (post-migration reconcile): rename the ExternalSecrets to `memorystore-valkey-auth-v2` / `memorystore-valkey-tls-v3`, repoint `global.redis` at them, remove the `-v1` entries, and set the real Valkey IP. **Folded into MR 3 (3058)** rather than raised separately — 3058 now carries the full reconcile (real IP + rename), matching the live in-cluster override: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3058. See Post-migration cleanup. **Infrastructure (Terraform, config-mgmt on ops.gitlab.net):** - [config-mgmt MR 15068](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15068) — comments out / removes the (test) Valkey instance. Once merged and applied, no Valkey exists until the change window recreates it. - Uncomment the `ops-gitlab-valkey` instance and the Vault CA resource (`gcs_source` set, so creation restores from the dump). Reviewed and approved, left in **Draft** until the window: [config-mgmt MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) - **Applied manually from the branch in-window** (maintenance mode blocks merge + CI), then **merged immediately after maintenance mode is lifted** to close the state/config divergence. - Post-migration: re-enable deletion protection on the instance: _TBD_ ## Change Details 1. **Services Impacted** - ~"Service::OpsGitlabNet" 2. **Change Technician** - `@gsgl` 3. **Change Reviewer** - @jomartin1 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-08-23 23:30 5. **Time tracking** - ~60–90 min (maintenance window: quiesce, snapshot, instance create + restore ~15 min, cutover, verify) 6. **Downtime Component** - `ops.gitlab.net` in maintenance mode for the duration of the window (writes blocked; Sidekiq paused). ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist <!--To be filled out by the reviewer.--> ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The change plan is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable rollback plan. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist - [x] The Change Criticality has been set appropriately and requirements have been reviewed. - [x] The change plan is technically accurate. - [x] The rollback plan is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [ ] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] An MR approver is confirmed available **during the change window**, although MR 3 approval/merge now happens post-window per the mitigation above rather than in-window. - [x] The change execution window respects the Production Change Lock periods. - [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [x] For ~C1 and ~C2 change issues, the change event is added to the GitLab Production calendar by the change-scheduler bot. - [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process. - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed at least 2 weeks in advance. ### Pre-change Checklist - [x] `rediss` + CA + auth path validated against a test Valkey instance (`PING → PONG` from the toolbox) and by `helm template`. The test instance is removed by [config-mgmt MR 15068](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15068), so no Valkey exists until the window; real connectivity is proven in-window once Sidekiq runs on Valkey. - [x] MR 1 (ExternalSecrets) reviewed, approved, and **merged ahead of the window**; CA + token Secrets confirmed synced in-cluster (autosync applied them). - [x] MR 2 (disable autosync) reviewed, approved, and **merged ahead of the window**; `gitlab-ops` autosync confirmed disabled. Keep the lead time short: while autosync is off, other `gitlab-ops` changes will not auto-apply until it is re-enabled. - [x] MR 3 (cutover) prepared as a **Draft** for early structural review. It is **not merged before or during the window** (see "Known constraint" above): `global.redis.host` (the Valkey IP) and both `memorystore-valkey-*` ExternalSecret `version:`s are only known once Terraform creates the instance and restores the dump in-window, and the gitlab.com → ops.gitlab.net mirror push cannot land while maintenance mode is on. The real values are applied in-window via an ArgoCD Application Helm parameter override instead; MR 3 is finalized, reviewed, and merged **after** the window, once the mirror is reachable again. - [x] The config-mgmt MR uncommenting the `ops-gitlab-valkey` instance and adding the Vault resource ([config-mgmt MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078)) is reviewed and approved, left as **Draft** until the window. It is applied manually in-window and merged immediately after maintenance mode is lifted. - [ ] **Check out the [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) branch locally and run `terraform init` before the window.** The ops environment sources modules from the ops.gitlab.net Terraform module registry (`ops.gitlab.net/gitlab-com/bootstrap/google`, `tcp-lb/google`, `generic-sv-with-group/google`, `storage-buckets/google`). Registry reads should survive maintenance mode, but pre-caching `.terraform/` removes the risk entirely, so the in-window run is only `plan` + `apply`. - [ ] **Confirm a config-mgmt approver is available shortly after the window ends** to review and merge [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) promptly (see the divergence warning in "Known constraint"). - [ ] **Note the current Vault versions before the window** for `env/ops/ns/gitlab/memorystore-valkey/auth` and `…/tls`, so the in-window bump to v2 / v3 can be confirmed rather than assumed. If the pre-window versions are not 1 and 2 respectively, adjust the override values in the cutover step accordingly. - [x] GCS bucket/path prepared for the Redis dump; Memorystore Valkey import-from-GCS path confirmed for the instance. - [x] Dry-run of the backup/restore into Valkey completed and verified (restored key counts match the source); results noted in a comment (see dry-run steps). - [x] The rollback plan is technically accurate and executable by anyone with access. - [x] Confirm the override target **and** the live autosync state before the window. `gitlab-ops--1a387` is **generated by the `service-gitlab-ops` ApplicationSet**, so the override goes on the ApplicationSet **template**, not the Application (which gets reconciled back — Execution notes issue 1). Check for pre-existing `helm.parameters` on the template's `sources[0]` (`gitlab` chart) and `sources[2]` (`vault-secrets` chart): `kubectl get applicationset service-gitlab-ops -n argocd -o jsonpath='{range .spec.template.spec.sources[*]}{.chart}{" -> "}{.helm.parameters}{"\n"}{end}'`. **Also confirm autosync is actually disabled on the live Application** — do not trust that MR 2 disabled it via git; the ApplicationSet's `ignoreApplicationDifferences` on `/spec/syncPolicy/automated/enabled` prevents a template change from propagating (Execution notes issue 2). `kubectl get application gitlab-ops--1a387 -n argocd -o jsonpath='{.spec.syncPolicy.automated.enabled}'` must print `false`; if it prints `true`, disable it by patching the Application directly before proceeding, or the template override will auto-deploy the cutover. ### Dry-run: Redis backup/restore rehearsal Rehearse the data-migration mechanism end to end before the window. It is non-destructive to production: it takes a `BGSAVE` snapshot of the in-cluster Redis (read-only) and restores it into a throwaway Valkey instance, which is then torn down. It validates the procedure (snapshot, GCS upload, `gcs_source` restore, key-count parity), not the final data (the real cutover takes a fresh, quiesced snapshot). - [x] Trigger a `BGSAVE` on the master (AUTH read in-pod): `kubectl exec -it ops-gitlab-redis-master-0 -n gitlab -c redis -- bash -c 'export REDISCLI_AUTH=$(cat /opt/bitnami/redis/secrets/redis-password) && redis-cli BGSAVE'`. Run during low traffic (BGSAVE forks the process). - [x] Poll until the save completes (`rdb_bgsave_in_progress:0`): `kubectl exec -it ops-gitlab-redis-master-0 -n gitlab -c redis -- bash -c 'export REDISCLI_AUTH=$(cat /opt/bitnami/redis/secrets/redis-password) && redis-cli INFO persistence | grep rdb_bgsave_in_progress'` - [x] Note the source key count for comparison: `kubectl exec -it ops-gitlab-redis-master-0 -n gitlab -c redis -- bash -c 'export REDISCLI_AUTH=$(cat /opt/bitnami/redis/secrets/redis-password) && redis-cli DBSIZE && redis-cli INFO keyspace'` - [x] Copy the dump off the pod: `kubectl cp gitlab/ops-gitlab-redis-master-0:/data/dump.rdb ./ops-gitlab-net-dump.rdb -c redis` - [x] Upload it to the migration bucket: `gsutil cp ops-gitlab-net-dump.rdb gs://gitlab-ops-redis-valkey-migration/` - [x] Rehearse the restore the way the cutover does it: create a throwaway Valkey instance from the uploaded dump (Terraform / `gcloud` with `gcs_source`; ~15 min). A freshly created instance starts empty and loads the dump, so the restored dataset is exactly the dump (no merge to reason about). - [x] Verify the restore over TLS + auth: connect and compare `DBSIZE` / `INFO keyspace` against the source counts above; spot-check sample keys. Confirm the create/restore operation reported success. - [x] Tear down the throwaway instance and remove the rehearsal dump from GCS so nothing lingers before the real cutover. - [x] Note the outcome (timings, key counts) in a comment on this issue. ## Detailed steps for the change ### Pre-execution steps - [x] Confirm there are no active ~severity::1 / ~severity::2 incidents. - [x] Confirm the config-mgmt MR (uncomment `ops-gitlab-valkey` + Vault resource, [config-mgmt MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078)) is approved and ready to apply. Connectivity cannot be pre-checked: no Valkey exists until this MR is applied in-window. - [ ] Confirm the CA + token Secrets are present in-cluster. - [ ] Confirm the [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) branch is checked out locally with `terraform init` already run (see Pre-change Checklist). ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - ~60–90 (Valkey instance create + restore is ~15 min of this) - [x] Set label ~"change::in-progress". - [x] Confirm the pre-window prerequisites are in place: MR 1 merged (ExternalSecrets synced in-cluster) and MR 2 merged (which disables `gitlab-ops` autosync). - [x] **Enable GitLab maintenance mode** (blocks user writes): log in to `ops.gitlab.net` as an admin, go to **Admin Area > Settings > General**, expand the **Maintenance Mode** section, toggle it on (optionally set a banner message), and **Save changes**. - [x] **Quiesce Sidekiq** for a consistent snapshot. Scaling the Deployment to 0 alone won't hold — the native Kubernetes HPA (`minReplicas: 3`), independent of ArgoCD, scales it back up. Delete (or zero) the HPA, then scale the Deployment to 0, and confirm no active jobs. With autosync off, ArgoCD will not restore the HPA until the cutover Sync. The `cache` DB can be skipped; `queues` and `shared_state` are what matters. - [x] **Snapshot the in-cluster Redis and upload to GCS** (same commands as the dry-run): `BGSAVE`, poll `rdb_bgsave_in_progress:0`, `kubectl cp gitlab/ops-gitlab-redis-master-0:/data/dump.rdb ./ops-gitlab-net-dump.rdb -c redis`, then `gsutil cp ops-gitlab-net-dump.rdb gs://gitlab-ops-redis-valkey-migration/`. - [x] **Create the Valkey instance via a manual Terraform apply from the [MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) branch.** The MR cannot be marked ready, merged, or run through CI while ops.gitlab.net is in maintenance mode (see "Known constraint"). The GCS backend is unaffected, so apply locally from the branch: ```bash cd environments/ops terraform plan -target=google_memorystore_instance.gitlab_valkey \ -target=vault_kv_secret_v2.gitlab_valkey_ca terraform apply -target=google_memorystore_instance.gitlab_valkey \ -target=vault_kv_secret_v2.gitlab_valkey_ca ``` Targeting keeps the blast radius to the two new resources and avoids picking up unrelated drift mid-window. Read the plan before applying. `gcs_source` restores from the dump uploaded in the previous step (the dump must already be in GCS). Wait for create + restore to finish (~15 min) and note the instance **IP**. The same apply writes the new CA to `env/ops/ns/gitlab/memorystore-valkey/tls` via the `vault_kv_secret_v2` resource (Vault/OpenBao is on GKE, unaffected by maintenance mode). - [x] **Update the auth token in Vault:** `gcloud beta memorystore instances token-auth-users auth-tokens list --token-auth-user=default --instance=ops-gitlab-valkey --location=us-central1 --project=gitlab-ops --format="value(token)"`, write it to `env/ops/ns/gitlab/memorystore-valkey/auth`. - [x] **Confirm the new Vault versions.** Expected: `env/ops/ns/gitlab/memorystore-valkey/auth` → **v2** (written in the previous step) and `env/ops/ns/gitlab/memorystore-valkey/tls` → **v3** (written by the Terraform apply). Read them back rather than assuming, and use whatever the actual versions are in the patch below. - [x] **Apply the cutover via ArgoCD override — on the ApplicationSet template, not the Application.** MR 3 is not merged in-window (see "Known constraint"), so the real values are applied as Helm parameter overrides. **`gitlab-ops--1a387` is generated by the `service-gitlab-ops` ApplicationSet (`applicationsSync: create-update`), so patching the Application directly is reverted within seconds — the override must go on the ApplicationSet template** (`/spec/template/spec/sources/N/helm/parameters`; Execution notes issues 1–2). It is multi-source: `global.redis.*` belongs to `sources[0]` (the `gitlab` chart, `releaseName: ops-gitlab`) and the `externalSecrets.*` version bumps to `sources[2]` (the `vault-secrets` chart, `releaseName: ops-gitlab-secrets`). Confirm the template source indices first (`kubectl get applicationset service-gitlab-ops -n argocd -o jsonpath='{range .spec.template.spec.sources[*]}{.chart}{" | "}{.repoURL}{"\n"}{end}'`) and re-confirm autosync is OFF (patching the template changes desired state and will auto-deploy if autosync is on). > **Note the deliberate naming deviation:** the ExternalSecret resources keep their existing `…-v1` names here and only their pinned `remoteRef.version` is bumped. A Secret named `memorystore-valkey-auth-v1` will hold Vault **v2** content, and `memorystore-valkey-tls-v1` will hold Vault **v3** content, until the post-migration rename MR. This is intentional; renaming the resources would require a `values-vault-secrets.yaml` change that cannot reach ArgoCD while maintenance mode is on. ```bash kubectl patch applicationset service-gitlab-ops -n argocd --type json -p '[ {"op":"add","path":"/spec/template/spec/sources/0/helm/parameters","value":[ {"name":"global.redis.host","value":"10.253.11.14"}, {"name":"global.redis.port","value":"6379"}, {"name":"global.redis.scheme","value":"rediss"}, {"name":"global.redis.redisTLS.caFile.secret","value":"memorystore-valkey-tls-v1"}, {"name":"global.redis.redisTLS.caFile.key","value":"ca.crt"}, {"name":"global.redis.auth.secret","value":"memorystore-valkey-auth-v1"}, {"name":"global.redis.auth.key","value":"token"} ]}, {"op":"add","path":"/spec/template/spec/sources/2/helm/parameters","value":[ {"name":"externalSecrets.memorystore-valkey-auth-v1.data[0].remoteRef.version","value":"2","forceString":true}, {"name":"externalSecrets.memorystore-valkey-tls-v1.data[0].remoteRef.version","value":"3","forceString":true} ]} ]' ``` The ApplicationSet controller renders these into the generated Application within a few seconds. `add` on an existing `parameters` path replaces that array wholesale, so fold in any pre-existing overrides found by the pre-change check. `forceString: true` keeps the two versions as quoted strings in the rendered `ExternalSecret`. **Verify they propagated into the generated app** (expect 7 params under `gitlab`, 2 under `vault-secrets`): `kubectl get application gitlab-ops--1a387 -n argocd -o jsonpath='{range .spec.sources[*]}{.chart}{"\n"}{range .helm.parameters[*]} {.name}={.value}{"\n"}{end}{end}'`. This in-cluster ApplicationSet edit persists through the window because `argocd-apps` has `selfHeal: false` and the git mirror is frozen; it is reconciled into git by MR 3 / MR 5 post-window and then removed (see Post-migration cleanup). - [x] **Sync.** If autosync is correctly disabled, manually **Sync** `gitlab-ops--1a387` in the ArgoCD UI to apply the `rediss` repoint, re-sync the ExternalSecrets, and recreate the Sidekiq HPA. **As executed:** autosync was in fact still on (Execution notes issue 2), so patching the ApplicationSet template auto-triggered this Sync immediately — before the Secret content was refreshed, which caused the stale-cred pod crashes. If autosync is on, expect the cutover to deploy the moment you patch the template. - [x] **Refresh the Secrets to the new Vault versions — the version bump alone will NOT do it.** With `refreshInterval: 0`, ESO writes each Secret **once at creation** and never reconciles it again: bumping `remoteRef.version` updates the ExternalSecret spec but **not** the underlying Secret content, and neither a `force-sync` annotation nor an ESO controller restart forces a re-pull (Execution notes issue 3). **Delete each target Secret so ESO recreates it** (`creationPolicy: Owner`) via the create path, pulling the pinned v2/v3. Do this **before** rolling the deployments, or pods pick up stale v1 creds. Do **not** print the secret material — compare `data-hash` instead: ```bash # before: note resourceVersion + data-hash kubectl -n gitlab get secret memorystore-valkey-auth-v1 memorystore-valkey-tls-v1 \ -o custom-columns='NAME:.metadata.name,RV:.metadata.resourceVersion,HASH:.metadata.annotations.reconcile\.external-secrets\.io/data-hash' # force a re-pull by deleting the target Secrets (ESO recreates within seconds) kubectl -n gitlab delete secret memorystore-valkey-auth-v1 memorystore-valkey-tls-v1 # after: confirm NEW data-hash, refreshTime=now, PINNED = 2 (auth) / 3 (tls) kubectl -n gitlab get externalsecret memorystore-valkey-auth-v1 memorystore-valkey-tls-v1 \ -o custom-columns='ES:.metadata.name,READY:.status.conditions[-1].status,REFRESH:.status.refreshTime,PINNED:.spec.data[0].remoteRef.version' kubectl -n gitlab get secret memorystore-valkey-auth-v1 memorystore-valkey-tls-v1 \ -o custom-columns='NAME:.metadata.name,RV:.metadata.resourceVersion,HASH:.metadata.annotations.reconcile\.external-secrets\.io/data-hash' ``` Safe because the Secret is ESO-owned (not ArgoCD-managed) and recreated in seconds. (The runbook's preferred convention — *renaming* the ExternalSecret to a new `-vN` object — avoids this entirely, because a new object always runs ESO's create path.) - [x] **Roll the GitLab deployments** so pods pick up the updated Secret contents: `kubectl -n gitlab rollout restart deploy -l release=ops-gitlab`. This is required, not optional: because the Secret names are unchanged, the pod specs are unchanged, so nothing restarts on its own. - [x] **Bring Sidekiq back — the HPA will NOT scale it up from 0.** The quiesce step scaled the Sidekiq Deployment to 0; recreating the HPA does not restore it (an HPA refuses to scale a target at 0 replicas — `scaling is disabled since the replica count of the target is zero` — and ArgoCD ignores `/spec/replicas` on HPA-managed Deployments). **Manually scale to `minReplicas`:** `kubectl -n gitlab scale deploy ops-gitlab-sidekiq-all-in-1-v2 --replicas=3`, then confirm the HPA goes `ScalingActive: True` and takes over (Execution notes issue 4). Sidekiq processing on Valkey (backlog draining, pods Ready with 0 restarts) is the primary proof the `rediss` + CA + auth path works end to end. - [x] **Shut down the in-cluster Redis (keep data):** scale its StatefulSet(s) to 0 (`kubectl -n gitlab get statefulset | grep redis`, then `kubectl -n gitlab scale statefulset <redis-sts…> --replicas=0`). The PVCs are retained, so it can be scaled back up for rollback; and if anything is still pointed at Redis it fails now, while we are watching, rather than silently. - [x] **Disable maintenance mode** (reverse of the enable step): in **Admin Area > Settings > General > Maintenance Mode**, toggle it off and **Save changes**. Only once Sidekiq is healthy and the in-cluster Redis is scaled down. - [x] **Merge [config-mgmt MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) now that writes to ops.gitlab.net work again.** This is the highest-priority post-maintenance-mode action: until it merges, the Terraform state and `main` disagree, and any ops-environment apply from `main` would destroy the live Valkey instance, its Vault CA secret, and the output. Steps: mark the MR ready, confirm the pipeline runs (CI is unblocked once maintenance mode is off), get the approval, merge, and **confirm the post-merge apply reports no changes** (the resources already exist in state from the manual apply, so a clean plan is the proof that state and config have converged). - [x] **Smoke test** `ops.gitlab.net` (login, git push/pull, CI, registry). With the in-cluster Redis down, a clean smoke test confirms GitLab is fully on Valkey. - [x] Leave ArgoCD autosync **off** through the soak, so the scaled-down Redis is not restored to `replicas: 1`. **Autosync must be disabled by patching the live Application directly** (`kubectl patch application gitlab-ops--1a387 -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"enabled":false}}}}'`) — a git/MR change to the ApplicationSet template does not propagate (Execution notes issue 2). Autosync is re-enabled as part of the post-migration teardown. - [x] Set label ~"change::complete". ### Post-migration cleanup > [!important] > > Until the first two items below are done, the live configuration is held in an ArgoCD `Application` override rather than in git, and two ExternalSecrets carry names that no longer match their pinned Vault versions. Treat these as the first follow-ups, not optional tidying. - [x] **Merge the reconcile MR — folded into MR 3 ([3058](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3058)), not a separate MR 5.** 3058 now covers both files, bringing git in line with what is live and fixing the naming at the same time: - `values-vault-secrets.yaml`: add `memorystore-valkey-auth-v2` (Vault path `…/memorystore-valkey/auth`, `property: token`, `version: "2"`, `secretKey: token`) and `memorystore-valkey-tls-v3` (Vault path `…/memorystore-valkey/tls`, `property: ca.crt`, `version: "3"`, `secretKey: ca.crt`), each copying the existing block's `refreshInterval: 0`, `secretStoreName: gitlab-secrets`, and `target` settings. Remove the `memorystore-valkey-auth-v1` and `memorystore-valkey-tls-v1` entries in the same MR. - `values.yaml`: set `global.redis.host` to the real Valkey IP (this supersedes MR 3, which can either be rebased into this MR or merged first and then amended), and point `global.redis.auth.secret` at `memorystore-valkey-auth-v2` and `global.redis.redisTLS.caFile.secret` at `memorystore-valkey-tls-v3`. - Confirm the mirror push to ops.gitlab.net lands after merge. - [x] **Sync and verify the rename took effect.** The new Secret names change the pod specs, so the rollout happens on its own this time. Confirm `memorystore-valkey-auth-v2` and `memorystore-valkey-tls-v3` exist and are `SecretSynced`, that the old `-v1` Secrets are gone (`deletionPolicy: Delete` removes them when the ExternalSecrets are pruned), and that Sidekiq and Rails stay healthy through the rollout. Watch for pods stuck mounting a Secret that no longer exists. - [x] **Remove the temporary `helm.parameters` overrides** from the **`service-gitlab-ops` ApplicationSet template** (that's where they live — not the Application; Execution notes issue 1). They're on `sources[0]` (gitlab chart) and `sources[2]` (vault-secrets chart): `kubectl patch applicationset service-gitlab-ops -n argocd --type json -p '[{"op":"remove","path":"/spec/template/spec/sources/0/helm/parameters"},{"op":"remove","path":"/spec/template/spec/sources/2/helm/parameters"}]'` (or `kubectl edit applicationset service-gitlab-ops -n argocd` and delete both blocks). This makes git the sole source of truth again. Do this only after the reconcile MR is merged and mirrored, then Sync and confirm the app is `Synced` with no diff. Leaving the overrides in place would silently shadow future `values.yaml` / `values-vault-secrets.yaml` changes to these same keys. - [x] Confirm [config-mgmt MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078) merged cleanly with a no-op apply (this is done in the change steps above; re-verify here if it was deferred for any reason). - [x] Tore down the in-cluster Redis once Valkey was confirmed good: merged the teardown MR ([3092](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3092): `redis.install: false` + remove `redis-secret-v1`) and synced with prune (selecting the redis StatefulSet). The sync pruned the StatefulSet but **not** its PVC — `redis-data-ops-gitlab-redis-master-0` was deleted manually (PV reclaim `Delete` → PD + old data destroyed); see Execution notes issue 6. Re-enabling autosync is tracked as an outstanding item below. > [!important] > > **Still outstanding after `~"change::complete"`** — minor Terraform tidy-up remaining (deletion protection is back on; these are cleanup, no risk): - [x] **Re-enabled ArgoCD autosync** on `gitlab-ops--1a387` (`enabled: true, prune: true, selfHeal: false`) by patching the Application directly (reverting MR 2 in git would not propagate — Execution notes issue 2). App confirmed `Synced`/`Healthy` afterward. - [x] Delete the RDB dump object from the migration bucket (`gsutil rm gs://gitlab-ops-redis-valkey-migration/ops-gitlab-net-dump.rdb`), then remove the bucket and any related resources it created (IAM bindings / service-account grants) from Terraform. The bucket is Terraform-managed, so drop the resources from config and apply rather than deleting by hand. Dump object deleted; bucket + custom role + service-agent grant + service-identity removal is in **config-mgmt MR [15167](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15167)** (Draft — pending plan review + apply). - [x] Re-enabled deletion protection on the `ops-gitlab-valkey` instance (Terraform), now that the cutover is confirmed. - [x] Remove `gcs_source` from the Valkey Terraform config — included in **config-mgmt MR [15167](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15167)**. Safe: `lifecycle.ignore_changes = ["gcs_source"]` (retained) means removing it will not trigger a recreate. - [x] **Confirmed the `lifecycle.ignore_changes = ["gcs_source"]` rule must stay permanently — it cannot be removed.** `gcs_source` is create-time-only (ForceNew) and still lives in the instance's state; with it removed from config, dropping the ignore rule makes Terraform plan to set it to null, which **forces a full replacement of the Valkey instance**. The rule is therefore retained for the life of the instance, with the comment updated to document this in config-mgmt MR [15168](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15168). (Confirms the caution flagged when this was deferred.) ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - ~30 > [!note] > > The cutover MR only repoints; it does not disable the subchart. After cutover the in-cluster Redis is scaled to 0, but its StatefulSet and PVCs remain, so its pre-cutover data is retained. Rollback is therefore fast (scale it back up and repoint), and it is fully removed only by the deliberate post-migration teardown. > > Since the cutover is applied via an ArgoCD **ApplicationSet template** Helm parameter override rather than a merged MR (see "Known constraint" and Execution notes issue 1), rollback during the window means **removing the override from the ApplicationSet template** (which reverts `gitlab-ops` to whatever `main` currently renders, i.e. the in-cluster Redis with the `-v1` ExternalSecrets pinned back to their original versions), not reverting a merged commit. - [ ] Scale the in-cluster Redis StatefulSet(s) back up (`--replicas=1`). - [ ] Remove the `helm.parameters` overrides added in the cutover step from the **`service-gitlab-ops` ApplicationSet template** (`kubectl patch applicationset service-gitlab-ops -n argocd --type json -p '[{"op":"remove","path":"/spec/template/spec/sources/0/helm/parameters"},{"op":"remove","path":"/spec/template/spec/sources/2/helm/parameters"}]'`, or `kubectl edit` and delete both blocks), so the app again renders from `main` (in-cluster Redis, ExternalSecrets back on their original pinned versions), and Sync. Note the ExternalSecrets will **not** revert their Secret content on their own (`refreshInterval: 0`); to roll back the creds, delete the `-v1` Secrets so ESO re-pulls the original pinned versions (Execution notes issue 3). - [ ] Roll the GitLab deployments (`kubectl -n gitlab rollout restart deploy -l release=ops-gitlab`). Required for the same reason as in the cutover: the Secret names never changed, so pods will not restart on their own when the contents revert. - [ ] Confirm GitLab is serving Redis from the in-cluster instance again (toolbox `Gitlab::Redis::SharedState … ping`). - [ ] Do **not** merge MR 3 / MR 5 or run the post-migration teardown MR (`redis.install: false` / `redis-secret-v1` removal) until Valkey is validated in production, so this rollback stays available. - [ ] The new Vault versions (auth v2, tls v3) can be left in place; nothing reads them once the override is removed, and Vault keeps the older versions the `-v1` ExternalSecrets are pinned to. - [ ] The freshly-created Valkey instance can be left idle or torn down (deletion protection is not re-enabled until post-migration). - [ ] **Decide what to do with [config-mgmt MR 15078](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15078).** The instance was created by a manual apply, so it exists in Terraform state while `main` still has it commented out. Either (a) merge 15078 anyway to keep state and config consistent and leave the instance idle, or (b) `terraform destroy -target=google_memorystore_instance.gitlab_valkey -target=vault_kv_secret_v2.gitlab_valkey_ca` and close the MR. **Do not simply abandon the MR without one of these**, or a later ops-environment apply from `main` will destroy the instance unexpectedly. - [ ] Set label ~"change::aborted". ## Monitoring ### Key metrics to observe - **Metric:** Sidekiq job error/failure & queue depth (ops) - **Location:** [ops.gitlab.net service overview](https://dashboards.gitlab.net/d/ops-gitlab-net-main/ops-gitlab-net-overview) (Sidekiq + Redis panels) - **What prompts a rollback:** jobs failing to enqueue/process, or Redis connection errors after cutover. - **Metric:** Rails/Puma error rate & apdex (web/api) on `ops.gitlab.net` - **Location:** [ops.gitlab.net service overview](https://dashboards.gitlab.net/d/ops-gitlab-net-main/ops-gitlab-net-overview) - **What prompts a rollback:** sustained 5xx or apdex drop tied to Redis connectivity. - **Metric:** Redis/Valkey connection errors in Rails logs (`getaddrinfo` / TLS / `NOAUTH`/`WRONGPASS`) - **Location:** [nonprod-log.gitlab.net](https://nonprod-log.gitlab.net) - **What prompts a rollback:** any TLS handshake, CA verification, or auth failures against the Valkey instance IP. - **Metric:** Memorystore Valkey server-side connections / CPU / memory - **Location (Grafana):** [memorystore-ops-gitlab-valkey overview](https://dashboards.gitlab.net/d/memorystore-ops-gitlab-valkey-main/memorystore-ops-gitlab-valkey3a-overview?orgId=1&from=2026-08-23T21:54:00.000Z&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-ops&var-environment=ops) - **Location (GCP Console):** [ops-gitlab-valkey instance](https://console.cloud.google.com/memorystore/valkey/locations/us-central1/instances/ops-gitlab-valkey/details?project=gitlab-ops) - **What prompts a rollback:** refused connections or saturation after cutover.