Skip to content

K8s Integration installs not working after trying to tie two projects to the same cluster.

Summary

Today I've tried to add our k8s cluster into Gitlab. When I tried for the first time, everything went smoothly, I've installed tiller into cluster from gitlab, I've installed prometheus from gitlab, everything ok so far.

Then I tried to connect second project to the same cluster - this is where things started to get messy Integration was okay - Tiller installed okay, however I was unable to install Prometheus

Operation failed. Check pod logs for install-prometheus for more details.

So running kubectl logs --namespace gitlab-managed-apps install-prometheus gives me this:

Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (30)...'
+ helm upgrade prometheus stable/prometheus --install --reset-values --tls --tls-ca-cert /data/helm/prometheus/config/ca.pem --tls-cert /data/helm/prometheus/config/cert.pem --tls-key /data/helm/prometheus/config/key.pem --version 6.7.3 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/prometheus/config/values.yaml
Retrying (30)...
Error: UPGRADE FAILED: remote error: tls: bad certificate

So I've removed the integrations from all clusters and did it again, only one project per cluster this time. Deleted gitlab-managed-apps namespace, delete RBACs, and did the integration and the tiller & prometheus installation again, but ever since I am stucked on this bad certificate error and I cannot install Prometheus from gitlab to any cluster from any project right now.

Steps to reproduce

  1. Create integration to kubernetes with one of your projects
  2. Install Tiller and Prometheus from gitlab
  3. Create integration to the same cluster with other project
  4. Install Tiller and Prometheus from gitlab - will fail
  5. Remove integration from both projects
  6. Create new integration from random project to this cluster
  7. Install Tiller
  8. Try to install Prometheus and observe the error

Example Project

Not dependant on project - can be triggered with empty project, just the cluster is needed (Google)

What is the current bug behavior?

Prometheus fails to install

What is the expected correct behavior?

Prometheus installed successfuly

Relevant logs and/or screenshots

Screenshot_from_2019-07-30_14-51-40

Results of GitLab application Check

This is on gitlab.com

Possible fixes

Honestly I havent figured yet out something more valuable, the certificates somehow seems expired or mismatched

Edited by Josef Korbel