Skip to content

Adding a grafana datasource to through configuration fails on secrets

In gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!8125 (merged), we added a Clickhouse Datasource as code through the Grafana configuration. Secrets were prepared in Vault.

During the first rollout attempt, secrets were created: https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/jobs/18889920. But the job got stuck after. A retry also got stuck: https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/jobs/18890500

We got notified by an alert that pods were failing to start on Slack: https://gitlab.slack.com/archives/C101F3796/p1750063795286829

The error in the GCP console was: CreateContainerConfigError,indicating that the secrets were not available.

image

@ahmadsherif jumped in and manually created those secrets, at which point the running, but stuck, deployment job completed: https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/jobs/18891901