Gitlab AutoDevOps promotheus installation fails after I had deleted a previous install

Hello together, I have setup gitlab kubernetes integration and run into a bug probably related to the way how prometheus was uninstalled from gitlab

Steps to reproduce

  • Install tiller and prometheus into kubernetes cluster (pods are installed in gitlab-managed-apps and everything is working fine)
  • uninstall prometheus
  • try to install prometheus again. Installation fails with the following log:
+ helm init --upgrade
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been upgraded to the current version.
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/prometheus/config/ca.pem --tls-cert /data/helm/prometheus/config/cert.pem --tls-key /data/helm/prometheus/config/key.pem
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
+ s=0
+ break
+ exit 0
+ helm upgrade prometheus stable/prometheus --install --reset-values --tls --tls-ca-cert /data/helm/prometheus/config/ca.pem --tls-cert /data/helm/prometheus/config/cert.pem --tls-key /data/helm/prometheus/config/key.pem --version 6.7.3 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/prometheus/config/values.yaml
UPGRADE FAILED
Error: "prometheus" has no deployed releases
Error: UPGRADE FAILED: "prometheus" has no deployed releases

I have seen such problems when installing pods myself. I assume a helm delete --purge is needed here.

Please help with workaround:

Unfortunately I have no idea how I can execute the purge from the command line myself. I have a basic tiller installation in kube-system. This seems to be the one my helm command is using.

I have red that I need to use a switch --tiller-namespace Unfortunately this does not work for me. I have cluster-admin privileges but caling

helm list --tiller-namespace gitlab-managed-apps

results in

Error: context deadline exceeded

Can somebody give me a hint on how I can execute the purge? I would realy like to reinstall prometheus again. Deleting gitlab-managed-apps did also not resolve the issue. Seems like tiller holds the configuration in a different place.

Thanks for your help!

Update: I found the description here, saying communication with the gitlab installed tiller is only possible with ssl

https://stackoverflow.com/questions/55588170/gitlab-installed-helm-error-context-deadline-exceeded

Can someone tell me how I can get to these certificates from my installation? The referenced link in the description is unfortunately dead.

Update 2: Workaround: helm delete --purge prometheus --tls --tls-ca-cert ca.crt.pem --tls-cert tls.crt.pem --tls-key tls.key.pem --tiller-namespace gitlab-managed-apps

certificates can be obtained from gitlab-managed-apps/tiller-secrets

Edited by cjohn001