Kubernetes Cluster Ingress & Knative install fails on self-hosted cluster
Summary
This is sort-of related to this issue but not exactly the same error message, hence this new issue.
I tried with multiple Kubernetes versions (1.15.7-1.17.x) with the same outcome.
This setup worked perfectly fine on an older version of Gitlab since I had it up and running late last year.
Helm Tiller and Prometheus installs fine on the cluster, so it is not a general cluster software installation problem.
Steps to reproduce
- Create a self-hosted Kubernetes cluster - I am using Kublr on vmware.
- CI / CD > Kubernetes > Add Kubernetes cluster
- Choose Add existing cluster
- Enter the following information:
- Kubernetes Cluster name:
- Environment scope: *
- API URL:
- CA-Certificate:
- Service-Token:
- check RBAC-enabled cluster
- check GitLab-managed cluster
- Applications > Helm Tiller > Install (successfully installed)
- Applications > Ingress > Install (fails)
- Applications > Knative > Install (fails)
What is the current bug behavior?
Ingress and Knative installations fail.
What is the expected correct behavior?
Ingress and Knative should be installed.
Relevant logs and/or screenshots
Pod logs from install-ingress
:
+ helm init --upgrade
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.3 .
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key /data/helm/ingress/config/key.pem
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
+ s=0
+ break
+ exit 0
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
+ helm upgrade ingress stable/nginx-ingress --install --atomic --cleanup-on-fail --reset-values --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key /data/helm/ingress/config/key.pem --version 1.29.7 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/ingress/config/values.yaml
Release "ingress" does not exist. Installing it now.
INSTALL FAILED
PURGING CHART
Error: release ingress failed: timed out waiting for the condition
Successfully purged a chart!
Error: release ingress failed: timed out waiting for the condition
Pod logs from ingress-nginx-ingress-controller
:
I0509 15:39:28.111677 7 flags.go:205] Watching for Ingress class: nginx
W0509 15:39:28.112177 7 flags.go:250] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0509 15:39:28.112245 7 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0509 15:39:28.112458 7 main.go:193] Creating API client for https://100.64.0.1:443
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.28.0
Build: git-1f93cb8f3
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.17.7
-------------------------------------------------------------------------------
I0509 15:39:28.129838 7 main.go:237] Running in Kubernetes cluster version v1.15 (v1.15.7) - git (clean) commit 6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4 - platform linux/amd64
I0509 15:39:28.138327 7 main.go:91] Validated gitlab-managed-apps/ingress-nginx-ingress-default-backend as the default backend.
I0509 15:39:28.365675 7 main.go:102] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
W0509 15:39:28.392765 7 store.go:636] Unexpected error reading configuration configmap: configmaps "ingress-nginx-ingress-controller" not found
I0509 15:39:28.407510 7 nginx.go:263] Starting NGINX Ingress controller
I0509 15:39:29.608018 7 nginx.go:307] Starting NGINX process
I0509 15:39:29.608039 7 leaderelection.go:242] attempting to acquire leader lease gitlab-managed-apps/ingress-controller-leader-nginx...
W0509 15:39:29.608567 7 controller.go:394] Service "gitlab-managed-apps/ingress-nginx-ingress-default-backend" does not have any active Endpoint
I0509 15:39:29.608764 7 controller.go:137] Configuration changes detected, backend reload required.
I0509 15:39:29.614192 7 status.go:86] new leader elected: ingress-nginx-ingress-controller-755849fcb4-vr276
I0509 15:39:29.681009 7 controller.go:153] Backend successfully reloaded.
I0509 15:39:29.681186 7 controller.go:162] Initial sync, sleeping for 1 second.
I0509 15:40:01.836855 7 leaderelection.go:252] successfully acquired lease gitlab-managed-apps/ingress-controller-leader-nginx
I0509 15:40:01.837105 7 status.go:86] new leader elected: ingress-nginx-ingress-controller-755849fcb4-62s2q
I0509 15:44:27.261264 7 main.go:152] Received SIGTERM, shutting down
I0509 15:44:27.261289 7 nginx.go:391] Shutting down controller queues
I0509 15:44:27.261310 7 status.go:117] updating status of Ingress rules (remove)
E0509 15:44:27.264697 7 status.go:121] error obtaining running IPs: []
I0509 15:44:27.264717 7 nginx.go:407] Stopping NGINX process
2020/05/09 15:44:27 [notice] 173#173: signal process started
I0509 15:44:30.294436 7 nginx.go:420] NGINX process has stopped
I0509 15:44:30.294463 7 main.go:160] Handled quit, awaiting Pod deletion
E0509 15:44:32.893388 7 leaderelection.go:331] error retrieving resource lock gitlab-managed-apps/ingress-controller-leader-nginx: Unauthorized
Pod logs from install-knative
:
+ helm init --upgrade
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.3 .
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/knative/config/ca.pem --tls-cert /data/helm/knative/config/cert.pem --tls-key /data/helm/knative/config/key.pem
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
+ s=0
+ break
+ exit 0
+ helm repo add knative https://storage.googleapis.com/triggermesh-charts
"knative" has been added to your repositories
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "knative" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
+ helm upgrade knative knative/knative --install --atomic --cleanup-on-fail --reset-values --tls --tls-ca-cert /data/helm/knative/config/ca.pem --tls-cert /data/helm/knative/config/cert.pem --tls-key /data/helm/knative/config/key.pem --version 0.9.0 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/knative/config/values.yaml
Release "knative" does not exist. Installing it now.
INSTALL FAILED
PURGING CHART
Error: release knative failed: timed out waiting for the condition
Successfully purged a chart!
Error: release knative failed: timed out waiting for the condition
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 8.11 Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.26.2 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 12.10.3 Revision: e01ceba1b49 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: https://gitlab.sanitized HTTP Clone URL: https://gitlab.sanitized/some-group/some-project.git SSH Clone URL: git@gitlab.sanitized:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 12.2.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 88 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/3 ... yes 5/6 ... yes 7/7 ... yes 7/8 ... yes 7/9 ... yes 7/10 ... yes 7/11 ... yes 7/12 ... yes 13/13 ... yes 7/16 ... yes 7/17 ... yes 13/19 ... yes 7/20 ... yes 16/21 ... yes 17/23 ... yes 17/25 ... yes 7/26 ... yes 19/27 ... yes 7/29 ... yes 7/30 ... yes 7/31 ... yes 7/32 ... yes 6/33 ... yes 20/34 ... yes 20/35 ... yes 20/36 ... yes 2/37 ... yes 20/38 ... yes 22/39 ... yes 23/40 ... yes 23/41 ... yes 23/42 ... yes 23/43 ... yes 24/46 ... yes 26/48 ... yes 27/49 ... yes 27/50 ... yes 23/51 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.26.2) Git user has default SSH configuration? ... yes Active users: ... 10 Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished