K8s Integration installs not working after trying to tie two projects to the same cluster.
Summary
Today I've tried to add our k8s cluster into Gitlab. When I tried for the first time, everything went smoothly, I've installed tiller into cluster from gitlab, I've installed prometheus from gitlab, everything ok so far.
Then I tried to connect second project to the same cluster - this is where things started to get messy Integration was okay - Tiller installed okay, however I was unable to install Prometheus
Operation failed. Check pod logs for install-prometheus for more details.
So running kubectl logs --namespace gitlab-managed-apps install-prometheus
gives me this:
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (30)...'
+ helm upgrade prometheus stable/prometheus --install --reset-values --tls --tls-ca-cert /data/helm/prometheus/config/ca.pem --tls-cert /data/helm/prometheus/config/cert.pem --tls-key /data/helm/prometheus/config/key.pem --version 6.7.3 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/prometheus/config/values.yaml
Retrying (30)...
Error: UPGRADE FAILED: remote error: tls: bad certificate
So I've removed the integrations from all clusters and did it again, only one project per cluster this time.
Deleted gitlab-managed-apps
namespace, delete RBACs, and did the integration and the tiller & prometheus installation again, but ever since I am stucked on this bad certificate error and I cannot install Prometheus from gitlab to any cluster from any project right now.
Steps to reproduce
- Create integration to kubernetes with one of your projects
- Install Tiller and Prometheus from gitlab
- Create integration to the same cluster with other project
- Install Tiller and Prometheus from gitlab - will fail
- Remove integration from both projects
- Create new integration from random project to this cluster
- Install Tiller
- Try to install Prometheus and observe the error
Example Project
Not dependant on project - can be triggered with empty project, just the cluster is needed (Google)
What is the current bug behavior?
Prometheus fails to install
What is the expected correct behavior?
Prometheus installed successfuly
Relevant logs and/or screenshots
Results of GitLab application Check
This is on gitlab.com
Possible fixes
Honestly I havent figured yet out something more valuable, the certificates somehow seems expired or mismatched