gitlab managed cert-manager uninstall completes but gitlab thinks uninstall timed out & believes it's still installed

Summary

cert manager was playing up (failing to renew one of the certs), so I decided to upgrade to the latest version by uninstalling+reinstalling.

The first install failed due to helm version issues (see #35686 (closed)) - the second time the uninstall completed successfully but gitlab believes the uninstall timed out and believes

Steps to reproduce

1. Install gitlab managed apps on gitlab.com in 2019
2. Attempt to uninstall cert-manager in May 2020.
3. Attempt the uninstall again a few minutes after it first fails

Example Project

https://gitlab.com/openid/conformance-suite/-/clusters/73043?tab=apps

What is the current bug behavior?

The uninstall happens successfully, but gitlab says it timed out.

What is the expected correct behavior?

cert-manager successfully uninstalls.

Relevant logs and/or screenshots

2020-05-18 13:18:30.845 BST + helm init --upgrade
2020-05-18 13:18:30.954 BST Creating /root/.helm
2020-05-18 13:18:30.954 BST Creating /root/.helm/repository
2020-05-18 13:18:30.954 BST Creating /root/.helm/repository/cache
2020-05-18 13:18:30.955 BST Creating /root/.helm/repository/local
2020-05-18 13:18:30.955 BST Creating /root/.helm/plugins
2020-05-18 13:18:30.955 BST Creating /root/.helm/starters
2020-05-18 13:18:30.955 BST Creating /root/.helm/cache/archive
2020-05-18 13:18:30.955 BST Creating /root/.helm/repository/repositories.yaml
2020-05-18 13:18:30.955 BST Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
2020-05-18 13:18:33.266 BST Adding local repo with URL: http://127.0.0.1:8879/charts
2020-05-18 13:18:33.268 BST $HELM_HOME has been configured at /root/.helm.
2020-05-18 13:18:33.300 BST
2020-05-18 13:18:33.300 BST Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.6 .
2020-05-18 13:18:33.315 BST + seq 1 30
2020-05-18 13:18:33.316 BST + helm version --tls --tls-ca-cert /data/helm/certmanager/config/ca.pem --tls-cert /data/helm/certmanager/config/cert.pem --tls-key /data/helm/certmanager/config/key.pem
2020-05-18 13:18:33.404 BST Client: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
2020-05-18 13:18:33.563 BST Server: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
2020-05-18 13:18:33.566 BST + s=0
2020-05-18 13:18:33.566 BST + break
2020-05-18 13:18:33.566 BST + exit 0
2020-05-18 13:18:33.567 BST + helm delete --purge certmanager --tls --tls-ca-cert /data/helm/certmanager/config/ca.pem --tls-cert /data/helm/certmanager/config/cert.pem --tls-key /data/helm/certmanager/config/key.pem
2020-05-18 13:18:33.808 BST [storage] 2020/05/18 12:18:33 getting release history for "certmanager"
2020-05-18 13:18:33.820 BST [tiller] 2020/05/18 12:18:33 uninstall: Deleting certmanager
2020-05-18 13:18:33.820 BST [tiller] 2020/05/18 12:18:33 executing 0 pre-delete hooks for certmanager
2020-05-18 13:18:33.820 BST [tiller] 2020/05/18 12:18:33 hooks complete for pre-delete certmanager
2020-05-18 13:18:33.820 BST [storage] 2020/05/18 12:18:33 updating release "certmanager.v1"
2020-05-18 13:18:34.254 BST [kube] 2020/05/18 12:18:34 Starting delete for "v1beta1.admission.certmanager.k8s.io" APIService
2020-05-18 13:18:34.261 BST cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled" "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.admission.certmanager.k8s.io"}
2020-05-18 13:18:34.265 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook" Service
2020-05-18 13:18:34.285 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cainjector" Deployment
2020-05-18 13:18:34.298 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook" Deployment
2020-05-18 13:18:34.315 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager" Deployment
2020-05-18 13:18:34.327 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook:webhook-authentication-reader" RoleBinding
2020-05-18 13:18:34.338 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cainjector" ClusterRoleBinding
2020-05-18 13:18:34.347 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook:auth-delegator" ClusterRoleBinding
2020-05-18 13:18:34.359 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-certificates" ClusterRoleBinding
2020-05-18 13:18:34.369 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-issuers" ClusterRoleBinding
2020-05-18 13:18:34.378 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-orders" ClusterRoleBinding
2020-05-18 13:18:34.389 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-challenges" ClusterRoleBinding
2020-05-18 13:18:34.403 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-ingress-shim" ClusterRoleBinding
2020-05-18 13:18:34.412 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-leaderelection" ClusterRoleBinding
2020-05-18 13:18:34.421 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-clusterissuers" ClusterRoleBinding
2020-05-18 13:18:34.430 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-issuers" ClusterRole
2020-05-18 13:18:34.446 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-edit" ClusterRole
2020-05-18 13:18:34.456 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-challenges" ClusterRole
2020-05-18 13:18:34.471 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-leaderelection" ClusterRole
2020-05-18 13:18:34.481 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cainjector" ClusterRole
2020-05-18 13:18:34.495 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook:webhook-requester" ClusterRole
2020-05-18 13:18:34.505 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-certificates" ClusterRole
2020-05-18 13:18:34.513 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-view" ClusterRole
2020-05-18 13:18:34.522 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-orders" ClusterRole
2020-05-18 13:18:34.545 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-clusterissuers" ClusterRole
2020-05-18 13:18:34.554 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager-controller-ingress-shim" ClusterRole
2020-05-18 13:18:34.564 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cainjector" ServiceAccount
2020-05-18 13:18:34.575 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook" ServiceAccount
2020-05-18 13:18:34.593 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-cert-manager" ServiceAccount
2020-05-18 13:18:34.602 BST 2020/05/18 12:18:34 uninstall: Failed deletion of "certmanager": no objects visited
2020-05-18 13:18:34.604 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook-ca" Certificate
2020-05-18 13:18:34.612 BST cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="gitlab-managed-apps/certmanager-webhook-ca"
2020-05-18 13:18:34.612 BST cert-manager/controller/certificates/certificates "level"=0 "msg"="certificate resource not found for key" "key"="gitlab-managed-apps/certmanager-webhook-ca"
2020-05-18 13:18:34.612 BST cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="gitlab-managed-apps/certmanager-webhook-ca"
2020-05-18 13:18:34.615 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook-webhook-tls" Certificate
2020-05-18 13:18:34.622 BST cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="gitlab-managed-apps/certmanager-webhook-webhook-tls"
2020-05-18 13:18:34.622 BST cert-manager/controller/certificates/certificates "level"=0 "msg"="certificate resource not found for key" "key"="gitlab-managed-apps/certmanager-webhook-webhook-tls"
2020-05-18 13:18:34.622 BST cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="gitlab-managed-apps/certmanager-webhook-webhook-tls"
2020-05-18 13:18:34.629 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook-ca" Issuer
2020-05-18 13:18:34.638 BST cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="gitlab-managed-apps/certmanager-webhook-ca"
2020-05-18 13:18:34.638 BST cert-manager/controller/issuers "msg"="issuer in work queue no longer exists" "error"="issuer.certmanager.k8s.io \"certmanager-webhook-ca\" not found"
2020-05-18 13:18:34.638 BST cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="gitlab-managed-apps/certmanager-webhook-ca"
2020-05-18 13:18:34.643 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook-selfsign" Issuer
2020-05-18 13:18:34.653 BST cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="gitlab-managed-apps/certmanager-webhook-selfsign"
2020-05-18 13:18:34.654 BST cert-manager/controller/issuers "msg"="issuer in work queue no longer exists" "error"="issuer.certmanager.k8s.io \"certmanager-webhook-selfsign\" not found"
2020-05-18 13:18:34.654 BST cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="gitlab-managed-apps/certmanager-webhook-selfsign"
2020-05-18 13:18:34.659 BST [kube] 2020/05/18 12:18:34 Starting delete for "certmanager-webhook" ValidatingWebhookConfiguration
2020-05-18 13:18:34.663 BST [tiller] 2020/05/18 12:18:34 error: release "certmanager": object "" not found, skipping delete
2020-05-18 13:18:34.663 BST [tiller] 2020/05/18 12:18:34 executing 0 post-delete hooks for certmanager
2020-05-18 13:18:34.663 BST [tiller] 2020/05/18 12:18:34 hooks complete for post-delete certmanager
2020-05-18 13:18:34.663 BST [tiller] 2020/05/18 12:18:34 purge requested for certmanager
2020-05-18 13:18:34.663 BST [storage] 2020/05/18 12:18:34 deleting release "certmanager.v1"
2020-05-18 13:18:34.665 BST cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled" "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"certmanager-webhook"}
2020-05-18 13:18:34.684 BST release "certmanager" deleted
2020-05-18 13:18:34.684 BST + kubectl delete secret -n gitlab-managed-apps letsencrypt-prod --ignore-not-found
2020-05-18 13:18:34.803 BST cert-manager/controller/certificates "level"=0 "msg"="shutting down queue as workqueue signaled shutdown"
2020-05-18 13:18:34.803 BST cert-manager/controller/orders "level"=0 "msg"="shutting down queue as workqueue signaled shutdown"
2020-05-18 13:18:34.803 BST cert-manager/controller/challenges "level"=0 "msg"="shutting down queue as workqueue signaled shutdown"
2020-05-18 13:18:34.803 BST cert-manager/controller/issuers "level"=0 "msg"="shutting down queue as workqueue signaled shutdown"
2020-05-18 13:18:34.803 BST cert-manager/metrics "level"=0 "msg"="stopping Prometheus metrics server..."
2020-05-18 13:18:34.803 BST cert-manager/controller/clusterissuers "level"=0 "msg"="shutting down queue as workqueue signaled shutdown"
2020-05-18 13:18:34.803 BST cert-manager/metrics "level"=0 "msg"="prometheus metrics server gracefully stopped"
2020-05-18 13:18:34.803 BST cert-manager/metrics "msg"="error running prometheus metrics server" "error"="http: Server closed" "address"="0.0.0.0:9402"
2020-05-18 13:18:34.804 BST cert-manager/controller/ingress-shim "level"=0 "msg"="shutting down queue as workqueue signaled shutdown"
2020-05-18 13:18:34.805 BST cert-manager/controller "level"=0 "msg"="control loops exited"
2020-05-18 13:18:35.156 BST secret "letsencrypt-prod" deleted
2020-05-18 13:18:35.181 BST + kubectl delete crd certificates.certmanager.k8s.io --ignore-not-found
2020-05-18 13:18:35.328 BST customresourcedefinition.apiextensions.k8s.io "certificates.certmanager.k8s.io" deleted
2020-05-18 13:18:35.379 BST + kubectl delete crd certificaterequests.certmanager.k8s.io --ignore-not-found
2020-05-18 13:18:35.480 BST customresourcedefinition.apiextensions.k8s.io "certificaterequests.certmanager.k8s.io" deleted
2020-05-18 13:18:35.510 BST + kubectl delete crd challenges.certmanager.k8s.io --ignore-not-found
2020-05-18 13:18:35.598 BST customresourcedefinition.apiextensions.k8s.io "challenges.certmanager.k8s.io" deleted

Output of checks

This bug happens on GitLab.com

Possible fixes

unknown. I seem totally stuck now, gitlab thinks cert-manager is installed, but it isn't - so attempting to uninstall it fails. gitlab doesn't offer the option to install cert-manager as it believes it is installed. There doesn't appear to be a way to tell gitlab that it's wrong.