[CI] Operator reconciling for `default` namespace instead of SHA-based namespace
The short commit SHA is used for the namespace, but the GitLab reconciler is working in the default namespace. This seems to be because there is a gitlab CR in the default namespace.
Example: https://gitlab.com/gitlab-org/gl-openshift/gitlab-operator/-/jobs/1093798469
/cc @dustinmm80 FYI
Controller logs
kube-rbac-proxy I0312 16:27:13.216908 1 main.go:186] Valid token audiences:
kube-rbac-proxy I0312 16:27:13.216991 1 main.go:232] Generating self signed cert as no cert is provided
kube-rbac-proxy I0312 16:27:13.494016 1 main.go:281] Starting TCP socket on 0.0.0.0:8443
kube-rbac-proxy I0312 16:27:13.494331 1 main.go:288] Listening securely on 0.0.0.0:8443
manager 2021-03-12T16:27:14.174Z INFO setup setting operator scope {"scope": "cluster"}
manager I0312 16:27:15.320830 1 request.go:621] Throttling request took 1.046952635s, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v1alpha1?timeout=32s
manager 2021-03-12T16:27:17.030Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
manager 2021-03-12T16:27:17.030Z INFO controller-runtime.builder skip registering a mutating webhook, admission.Defaulter interface is not implemented {"GVK": "apps.gitlab.com/v1beta1, Kind=GitLab"}
manager 2021-03-12T16:27:17.030Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.gitlab.com/v1beta1, Kind=GitLab", "path": "/validate-apps-gitlab-com-v1beta1-gitlab"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.webhook registering webhook {"path": "/validate-apps-gitlab-com-v1beta1-gitlab"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.builder skip registering a mutating webhook, admission.Defaulter interface is not implemented {"GVK": "apps.gitlab.com/v1beta1, Kind=Runner"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.gitlab.com/v1beta1, Kind=Runner", "path": "/validate-apps-gitlab-com-v1beta1-runner"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.webhook registering webhook {"path": "/validate-apps-gitlab-com-v1beta1-runner"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.gitlab.com/v1beta1, Kind=GLBackup", "path": "/mutate-apps-gitlab-com-v1beta1-glbackup"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-apps-gitlab-com-v1beta1-glbackup"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.gitlab.com/v1beta1, Kind=GLBackup", "path": "/validate-apps-gitlab-com-v1beta1-glbackup"}
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.webhook registering webhook {"path": "/validate-apps-gitlab-com-v1beta1-glbackup"}
manager 2021-03-12T16:27:17.031Z INFO setup starting manager
manager I0312 16:27:17.031664 1 leaderelection.go:242] attempting to acquire leader lease 52ff4c22/852d23b0.gitlab.com...
manager 2021-03-12T16:27:17.031Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
manager 2021-03-12T16:27:17.032Z INFO controller-runtime.webhook.webhooks starting webhook server
manager 2021-03-12T16:27:17.032Z INFO controller-runtime.certwatcher Updated current TLS certificate
manager 2021-03-12T16:27:17.032Z INFO controller-runtime.webhook serving webhook server {"host": "", "port": 9443}
manager 2021-03-12T16:27:17.032Z INFO controller-runtime.certwatcher Starting certificate watcher
manager I0312 16:27:17.073873 1 leaderelection.go:252] successfully acquired lease 52ff4c22/852d23b0.gitlab.com
manager 2021-03-12T16:27:17.074Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GLBackup", "controller": "glbackup", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:17.073Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"52ff4c22","name":"852d23b0.gitlab.com","uid":"1d37c5ca-0dd9-48ee-91a1-9f2c73d46204","apiVersion":"v1","resourceVersion":"19345527"}, "reason": "LeaderElection", "message": "gitlab-controller-manager-655fcc5544-n9xdp_ea692c29-d460-469b-a2dc-cece7bf3dc16 became leader"}
manager 2021-03-12T16:27:17.074Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:17.074Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:17.174Z INFO controller Starting Controller {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GLBackup", "controller": "glbackup"}
manager 2021-03-12T16:27:17.174Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:17.274Z INFO controller Starting workers {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GLBackup", "controller": "glbackup", "worker count": 1}
manager 2021-03-12T16:27:17.275Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:20.271Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/validate-apps-gitlab-com-v1beta1-gitlab", "UID": "a24a83a9-8d18-467d-90dd-633b4f226ed8", "kind": "apps.gitlab.com/v1beta1, Kind=GitLab", "resource": {"group":"apps.gitlab.com","version":"v1beta1","resource":"gitlabs"}}
manager 2021-03-12T16:27:20.274Z INFO gitlab-resource validate create {"name": "gitlab"}
manager 2021-03-12T16:27:20.274Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/validate-apps-gitlab-com-v1beta1-gitlab", "code": 200, "reason": "", "UID": "a24a83a9-8d18-467d-90dd-633b4f226ed8", "allowed": true}
manager 2021-03-12T16:27:20.676Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:20.678Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:20.778Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:21.079Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:21.263Z INFO controller Starting Controller {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner"}
manager 2021-03-12T16:27:22.364Z INFO controller Starting workers {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner", "worker count": 1}
manager 2021-03-12T16:27:22.377Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.378Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.478Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.478Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.579Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.679Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.780Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.780Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.880Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:22.981Z INFO controller Starting EventSource {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "source": "kind source: /, Kind="}
manager 2021-03-12T16:27:23.081Z INFO controller Starting Controller {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab"}
manager 2021-03-12T16:27:23.081Z INFO controller Starting workers {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab", "worker count": 1}
manager 2021-03-12T16:27:23.081Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "default/gitlab", "name": "gitlab", "namespace": "default"}
manager 2021-03-12T16:27:23.283Z INFO template Rendering a new template. {"namespace": "default", "releaseName": "gitlab", "hash": "9ff3284388c76e87"}
manager 2021-03-12T16:27:23.283Z DEBUG template Looking for the designated GitLab Chart in the specified directory. {"chartVersion": "4.9.1", "directory": "/charts"}
manager 2021-03-12T16:27:47.075Z DEBUG template The template is rendered. Check the warnings (if any). {"namespace": "default", "releaseName": "gitlab", "hash": "9ff3284388c76e87", "warnings": 0}
manager 2021-03-12T16:27:47.075Z DEBUG template Caching the template. {"namespace": "default", "releaseName": "gitlab", "hash": "9ff3284388c76e87"}
manager 2021-03-12T16:27:47.075Z DEBUG template Retrieving the cached template {"namespace": "default", "releaseName": "gitlab", "hash": "9ff3284388c76e87"}
manager 2021-03-12T16:27:47.075Z DEBUG controllers.GitLab Ensuring Job's ConfigMap exists {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default", "configmap": "gitlab-shared-secrets"}
manager 2021-03-12T16:27:47.075Z DEBUG controllers.GitLab Creating Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:27:47.106Z INFO controllers.GitLab Waiting for Job to finish {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:27:47.106Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:27:50.106Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:27:53.106Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:27:56.107Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:27:59.107Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:02.107Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:05.107Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:08.108Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:11.108Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:14.108Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:17.109Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:20.109Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:23.109Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:26.109Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:29.110Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:32.110Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:35.110Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:38.110Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:41.111Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:44.111Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:47.111Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:50.112Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:53.112Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:56.112Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:28:59.112Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:02.113Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:05.113Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:08.113Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:11.113Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:14.114Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:17.114Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:20.114Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:23.115Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:26.115Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:29.115Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:32.116Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:35.116Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:38.116Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager E0312 16:29:40.605008 1 leaderelection.go:324] error initially creating leader election record: configmaps "852d23b0.gitlab.com" is forbidden: unable to create new content in namespace 52ff4c22 because it is being terminated
manager 2021-03-12T16:29:41.116Z DEBUG controllers.GitLab Checking the status of Job {"gitlab": "gitlab.default", "job": "gitlab-shared-secrets-1-yq0", "namespace": "default"}
manager 2021-03-12T16:29:41.271Z INFO controller Stopping workers {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "Runner", "controller": "runner"}
manager 2021-03-12T16:29:41.272Z INFO controller Stopping workers {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GLBackup", "controller": "glbackup"}
manager 2021-03-12T16:29:41.272Z INFO controller Stopping workers {"reconcilerGroup": "apps.gitlab.com", "reconcilerKind": "GitLab", "controller": "gitlab"}
kube-rbac-proxy I0312 16:29:41.273052 1 main.go:343] received interrupt, shutting down
kube-rbac-proxy E0312 16:29:41.273153 1 main.go:296] failed to gracefully close secure listener: close tcp [::]:8443: use of closed network connection
manager 2021-03-12T16:29:41.272Z INFO controller-runtime.webhook shutting down webhook server
manager stream closed
kube-rbac-proxy stream closedEdited by Mitchell Nielsen