Reconciliation in an endless loop of creating and deleting serviceMonitors
Summary
After upgrading to the latest operator and helm chart the Operator gets stuck in a loop of repeatedly creating and then immediately deleting serviceMonitor resources for gitlab-exporter
, gitaly
, redis
and gitlab-webservice
.
Steps to reproduce
Update to operator 0.23.2, then update the instance to chart 7.3.2
Configuration used
I can provide specific keys in the helm values as needed (it is rather large), I currently do not see immediately relevant values in it.
Current behavior
the logs have an endless list of this
2023-09-08T11:24:03Z INFO controllers.GitLab Reconciling GitLab {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}}
2023-09-08T11:24:03Z DEBUG controllers.GitLab version information {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "upgrade": false, "current version": "7.3.2", "desired version": "7.3.2"}
2023-09-08T11:24:04Z INFO controllers.GitLab running all migrations {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}}
2023-09-08T11:24:04Z INFO controllers.GitLab reconciling Webservice Deployments {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "pause": false}
2023-09-08T11:24:04Z INFO controllers.GitLab reconciling Sidekiq Deployments {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "pause": false}
2023-09-08T11:24:04Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "type": "*v1.ServiceMonitor", "reference": {"name":"gitlab-gitlab-exporter","namespace":"gitlab-system"}, "outcome": "created"}
2023-09-08T11:24:04Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "type": "*v1.ServiceMonitor", "reference": {"name":"gitlab-gitaly","namespace":"gitlab-system"}, "outcome": "created"}
2023-09-08T11:24:04Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "type": "*v1.ServiceMonitor", "reference": {"name":"gitlab-webservice","namespace":"gitlab-system"}, "outcome": "created"}
2023-09-08T11:24:04Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "type": "*v1.ServiceMonitor", "reference": {"name":"gitlab-redis","namespace":"gitlab-system"}, "outcome": "created"}
2023-09-08T11:24:04Z INFO controllers.GitLab Object deleted {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "monitoring.coreos.com/v1, Kind=ServiceMonitor", "name": "gitlab-gitlab-exporter"}
2023-09-08T11:24:04Z INFO controllers.GitLab Object deleted {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "monitoring.coreos.com/v1, Kind=ServiceMonitor", "name": "gitlab-gitaly"}
2023-09-08T11:24:04Z INFO controllers.GitLab Object deleted {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "monitoring.coreos.com/v1, Kind=ServiceMonitor", "name": "gitlab-webservice"}
2023-09-08T11:24:04Z INFO controllers.GitLab Object deleted {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "monitoring.coreos.com/v1, Kind=ServiceMonitor", "name": "gitlab-redis"}
and kubectl shows that indeed these resources are constantly created/destroyed.
The gitlab CR is okay
NAME STATUS VERSION
gitlab Running 7.3.2
and the application itself seems to be behaving properly
Expected behavior
The operator not getting stuck in a loop
Versions
- Operator: 0.23.2
- Platform: EKS
- Kubernetes:
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:48Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"darwin/arm64"}
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:48Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.4-eks-2d98532", GitCommit:"3d90c097c72493c2f1a9dd641e4a22d24d15be68", GitTreeState:"clean", BuildDate:"2023-07-28T16:51:44Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Relevant logs
see the current behavior part