2022-12-01: deploy stuck on gstg GKE

Current Status

We had several Pods on CrashLoopBackOff status during a gstg deployment:

gitlab-webservice-api-695dfffc69-cd62w             3/3     Running            0                101m
gitlab-webservice-api-695dfffc69-sjnx8             3/3     Running            0                102m
gitlab-webservice-api-695dfffc69-tnsjg             2/3     CrashLoopBackOff   13 (3m27s ago)   54m
gitlab-webservice-api-86b94f7c45-4lwl4             2/3     CrashLoopBackOff   15 (21s ago)     58m
gitlab-webservice-git-55c9968888-gkcqt             2/3     CrashLoopBackOff   14 (29s ago)     58m
gitlab-webservice-git-66fb766968-2w9gl             3/3     Running            0                99m
gitlab-webservice-git-66fb766968-bzlsz             2/3     Running            7 (7m48s ago)    18m
gitlab-webservice-git-66fb766968-l6ss5             2/3     CrashLoopBackOff   6 (4m32s ago)    17m
gitlab-webservice-git-66fb766968-qxsld             3/3     Running            0                102m
gitlab-webservice-web-75c77bfd8-d7qm4              3/3     Running            0                99m
gitlab-webservice-web-75c77bfd8-dh56f              2/3     CrashLoopBackOff   8 (3m51s ago)    23m
gitlab-webservice-web-75c77bfd8-dwxb8              2/3     Running            9 (5m15s ago)    24m
gitlab-webservice-web-75c77bfd8-g95l8              3/3     Running            0                101m
gitlab-webservice-web-75c77bfd8-ndhrf              3/3     Running            0                98m
gitlab-webservice-web-75c77bfd8-vjb5k              3/3     Running            0                97m
gitlab-webservice-web-7b6fd77d68-4b9fg             2/3     CrashLoopBackOff   8 (2m10s ago)    23m
gitlab-webservice-web-7b6fd77d68-x5wmp             2/3     CrashLoopBackOff   15 (62s ago)     58m
gitlab-webservice-websockets-76cf4df6dd-gdgws      3/3     Running            0                100m
gitlab-webservice-websockets-76cf4df6dd-rbnks      3/3     Running            0                102m
gitlab-webservice-websockets-796dd7b7d8-47647      2/3     CrashLoopBackOff   14 (2m57s ago)   58m

This was caused by the gitlab_metrics_error_rate_sli feature flag which was enabled only in gstg.

The feature flag has now been disabled, and we are no longer experiencing Pod Crashloops.

More information will be added as we investigate the issue. For customers believed to be affected by this incident, please subscribe to this issue or monitor our status page for further updates.

📚 References and helpful links

Recent Events (available internally only):

• Deployments ❙ Feature Flag Changes ❙ Gitlab.com Latest Updates ❙ Runbook for Release Rollback
• Infrastructure Configurations
• GCP Events (e.g. host failure)

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

• Corrective action ❙ Infradev
• Incident Review ❙ Infra investigation followup
• Confidential Support contact ❙ QA investigation

---

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.