Controller tries to delete HPAs with wrong API version
Summary
The Operator cleans up managed resources that are no longer needed. When a HorizontalPodAutoscaler (HPA) is cleaned up, the deletion targets multiple API versions, resulting in failures for APIs for which the HPA was not created.
On K8s 1.25 (kind):
2024-01-24T15:13:17Z INFO controllers.GitLab Object deleted {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "autoscaling/v2, Kind=HorizontalPodAutoscaler", "name": "gitlab-webservice-default"}
2024-01-24T15:13:17Z INFO controllers.GitLab Object deleted {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "autoscaling/v2, Kind=HorizontalPodAutoscaler", "name": "gitlab-sidekiq-all-in-1-v2"}
2024-01-24T15:13:17Z ERROR controllers.GitLab Could not delete the object {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "autoscaling/v2beta2, Kind=HorizontalPodAutoscaler", "name": "gitlab-webservice-default", "error": "horizontalpodautoscalers.autoscaling \"gitlab-webservice-default\" not found"}
gitlab.com/gitlab-org/cloud-native/gitlab-operator/controllers.(*GitLabReconciler).Reconcile
/workspace/controllers/gitlab_controller.go:469
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226
2024-01-24T15:13:17Z ERROR controllers.GitLab Could not delete the object {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "autoscaling/v2beta2, Kind=HorizontalPodAutoscaler", "name": "gitlab-sidekiq-all-in-1-v2", "error": "horizontalpodautoscalers.autoscaling \"gitlab-sidekiq-all-in-1-v2\" not found"}
gitlab.com/gitlab-org/cloud-native/gitlab-operator/controllers.(*GitLabReconciler).Reconcile
/workspace/controllers/gitlab_controller.go:469
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226
2024-01-24T15:13:17Z ERROR controllers.GitLab Could not delete the object {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "autoscaling/v1, Kind=HorizontalPodAutoscaler", "name": "gitlab-webservice-default", "error": "horizontalpodautoscalers.autoscaling \"gitlab-webservice-default\" not found"}
gitlab.com/gitlab-org/cloud-native/gitlab-operator/controllers.(*GitLabReconciler).Reconcile
/workspace/controllers/gitlab_controller.go:469
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226
2024-01-24T15:13:17Z ERROR controllers.GitLab Could not delete the object {"gitlab": {"name":"gitlab","namespace":"gitlab-system"}, "kind": "autoscaling/v1, Kind=HorizontalPodAutoscaler", "name": "gitlab-sidekiq-all-in-1-v2", "error": "horizontalpodautoscalers.autoscaling \"gitlab-sidekiq-all-in-1-v2\" not found"}
gitlab.com/gitlab-org/cloud-native/gitlab-operator/controllers.(*GitLabReconciler).Reconcile
/workspace/controllers/gitlab_controller.go:469
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226
How to reproduce
- Create a GitLab CR
- Confirm that the CR status is
Running
and that the webservice and sidekiq HPAs have been created - Disable webservice and sidekiq
gitlab.{webservice|sidekiq}.enabled=false
- Check the logs
Acceptance Criteria
The Operator should only attempt to delete HPAs with the API version, they were previously created with.
Edited by Clemens Beck