Skip to content

Operator deletes object with invalid k8s spec

Summary

While re-deploying our K8s Operator-based GitLab instance after modifying the helm chart (specifically, we added extra volumes to Gitaly), the operator removed the Gitaly statefulset because its k8s spec was invalid. The operator currently detects invalid k8s specs by manually decoding them (https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/helm/builder.go#L179), but then ignores these issues, printing them to debug logs (https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/controllers/gitlab/template.go#L58-59) and proceeding with the reconcilation. As a result, if one of the objects in the GitLab chart becomes invalid, the operator simply removes it. This caused an incident with our production GitLab instance.

Steps to reproduce

Make an error in the GitLab chart, for example, provide an invalid volume configuration via extraVolumes. Rollout the helm chart for the GitLab object. The operator will then delete the invalid object without reporting any error.

Configuration used

Current behavior

GitLab operator removes the object because it's missing from the target object configuration, and does not report any error.

Expected behavior

GitLab operator detects the error and stops the reconcilation.

Versions

  • Operator: 2.4.1
  • Platform:
    • Self-hosted: Kubernetes
  • Kubernetes: (kubectl version)
    • Client: v1.33.4
    • Server: v1.30.9

Relevant logs

2025-10-06T08:57:03.266005689Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.266005689Z stderr F {"level":"info","ts":1759741023.2659159,"logger":"controllers.GitLab","msg":"reconciling GitLab","gitlab":{"name":"gitlab","namespace":"gitlab"}}
2025-10-06T08:57:03.335205082Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.335205082Z stderr F {"level":"info","ts":1759741023.3351007,"logger":"controllers.GitLab","msg":"GitLab is initializing","gitlab":{"name":"gitlab","namespace":"gitlab"},"operation":"install","current version":"9.4.1","desired version":"9.4.1"}
2025-10-06T08:57:03.488524533Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.488524533Z stderr F {"level":"info","ts":1759741023.4884195,"logger":"controllers.GitLab","msg":"ensuring migrations Job has finished","gitlab":{"name":"gitlab","namespace":"gitlab"}}
2025-10-06T08:57:03.53420872Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.534208720Z stderr F {"level":"info","ts":1759741023.5341492,"logger":"controllers.GitLab","msg":"ensuring Webservice and Sidekiq are reconciled if enabled","gitlab":{"name":"gitlab","namespace":"gitlab"}}
2025-10-06T08:57:03.57258475Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.572584750Z stderr F {"level":"info","ts":1759741023.5723984,"logger":"controllers.GitLab","msg":"object deleted","gitlab":{"name":"gitlab","namespace":"gitlab"},"kind":"apps/v1, Kind=StatefulSet","name":"gitlab-gitaly-default"}
2025-10-06T08:57:03.633678024Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.633678024Z stderr F {"level":"info","ts":1759741023.6336124,"logger":"controllers.GitLab","msg":"object deleted","gitlab":{"name":"gitlab","namespace":"gitlab"},"kind":"batch/v1, Kind=Job","name":"gitlab-shared-secrets-22707ba"}
2025-10-06T08:57:03.639343915Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.639343915Z stderr F {"level":"info","ts":1759741023.6392825,"logger":"controllers.GitLab","msg":"object deleted","gitlab":{"name":"gitlab","namespace":"gitlab"},"kind":"batch/v1, Kind=Job","name":"gitlab-shared-secrets-22707ba-selfsign"}
2025-10-06T08:57:03.645506898Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.645506898Z stderr F {"level":"info","ts":1759741023.6454628,"logger":"controllers.GitLab","msg":"object deleted","gitlab":{"name":"gitlab","namespace":"gitlab"},"kind":"batch/v1, Kind=Job","name":"gitlab-migrations-5cadce3-ce-88"}
2025-10-06T08:57:03.662417169Z gitlab/gitlab-controller-manager-57df58f49d-dzrt7/manager 2025-10-06T08:57:03.662417169Z stderr F {"level":"info","ts":1759741023.6623535,"logger":"controllers.GitLab","msg":"GitLab is running and available to accept requests","gitlab":{"name":"gitlab","namespace":"gitlab"}}