2023-05-08: Chef client failures in multiple environments

Customer Impact

Current Status

Chef failures - https://thanos.gitlab.net/graph?g0.expr=sum%20by%20(env%2C%20type)%20(chef_client_error)%3E%200&g0.tab=0&g0.stacked=0&g0.range_input=2h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D

Status	Issue	Resolution
✅ Failures in pre env	Unable to retrieve secret 'chef/env/pre/shared/rsyslog_client' using role 'chef_pre_console' from Vault endpoint 'https://vault.ops.gke.gitlab.net'	https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/3420
✅ Failures on monitoring hosts	runbook clone v2.249.0 -> v2.249.0 (would clobber existing tag)	will delete v2.249.0 on .com and ops
✅ Failures on postgres VMs due 16.0 deprecations	#13136 (comment 1381458637)	Fix deprecations by running gitlab-ctl reconfigure
✅ Failures on testbed prometheus VM	unable to access host, environment is no longer used?	https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5722
❌ Failure on patroni-bs-test	Unable to access host, possibly needs to be restarted or removed	asked in slack
✅ Failure on runners-manager-saas-macos-staging-green-2	Chef purposely disabled on 2023-04-25 with the msg Disable chef until the next deployment	asked in slack we aren't sure why this is in a failed state but it should go away on its own after it is reenabled. ignoring for now

📚 References and helpful links

Recent Events (available internally only):

• Feature Flag Log - Chatops to toggle Feature Flags Documentation
• Infrastructure Configurations
• GCP Events (e.g. host failure)

Deployment Guidance

• Deployments Log | Gitlab.com Latest Updates
• Reach out to Release Managers for S1/S2 incidents to discuss Rollbacks, Hot Patching or speeding up deployments. | Rollback Runbook | Hot Patch Runbook

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

• Corrective action ❙ Infradev
• Incident Review ❙ Infra investigation followup
• Confidential Support contact ❙ QA investigation

---

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.