k8s integration: could not find a ready tiller pod
Summary
Trying to integrate a kubernetes cluster falls down when installing GitLab Runner.
Steps to reproduce
I added a new k8s cluster and installed Helm Tiller. Then I clicked on Install for GitLab Runner.
What is the current bug behavior?
The installation stalls for a very long time, and eventually shows
Something went wrong while installing GitLab Runner
Operation timed out. Check pod logs for install-runner for more detail
What is the expected correct behavior?
I should be able to install GitLab Runner.
Relevant logs and/or screenshots
Pods:
> kubectl get pods --namespace=gitlab-managed-apps
NAME READY STATUS RESTARTS AGE
install-runner 0/1 Error 0 3h18m
tiller-deploy-b575cf7dd-4kh75 1/1 Running 0 3h18m
Logs:
> kubectl logs install-runner --namespace=gitlab-managed-apps
+ helm init --upgrade
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.1 .
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/runner/config/ca.pem --tls-cert /data/helm/runner/config/cert.pem --tls-key /data/helm/runner/config/key.pem
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Error: could not find a ready tiller pod
+ s=1
+ sleep 1s
+ echo 'Retrying (1)...'
+ helm version --tls --tls-ca-cert /data/helm/runner/config/ca.pem --tls-cert /data/helm/runner/config/cert.pem --tls-key /data/helm/runner/config/key.pem
Retrying (1)...
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Error: could not find a ready tiller pod
Pod Info:
Expand for Pod Info
> kubectl describe pod install-runner --namespace=gitlab-managed-apps Name: install-runner Namespace: gitlab-managed-apps Priority: 0 Node: cloud-2/158.184.167.119 Start Time: Fri, 21 Feb 2020 16:25:03 +0000 Labels: gitlab.org/action=install gitlab.org/application=runner Annotations: cni.projectcalico.org/podIP: 10.42.1.224/32 Status: Failed IP: 10.42.1.224 IPs: IP: 10.42.1.224 Containers: helm: Container ID: docker://96cb7c13b3497a4df8111b7060f51fd883116f24f54a8ee09105c2c78fc8b20c Image: registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.16.1-kube-1.13.12 Image ID: docker-pullable://registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.16.1-kube-1.13.12@sha256:7bbe9ff6a60f4064f492c72e44ae107bc5 dc4778c4fffb35148a4ee376caf7ba Port: Host Port: Command: /bin/sh Args: -c $(COMMAND_SCRIPT) State: Terminated Reason: Error Exit Code: 1 Started: Fri, 21 Feb 2020 16:25:05 +0000 Finished: Fri, 21 Feb 2020 18:25:42 +0000 Ready: False Restart Count: 0 Environment: HELM_VERSION: 2.16.1 TILLER_NAMESPACE: gitlab-managed-apps COMMAND_SCRIPT: set -xeo pipefail helm init --upgrade for i in $(seq 1 30); do helm version --tls --tls-ca-cert /data/helm/runner/config/ca.pem --tls-cert /data/helm/runner/config/cert.pem --tls-key /data/helm/runner/config/key.pem && s=0 && break || s=$?; sleep 1s; echo "Retrying ($i)..."; done; (exit $s) helm repo add runner https://charts.gitlab.io helm repo update helm upgrade runner runner/gitlab-runner --install --reset-values --tls --tls-ca-cert /data/helm/runner/config/ca.pem --tls-cert /data/helm/runner/config/cert.pem --tls-key /data/helm/runner/config/key.pem --version 0.12.0 --set rbac.create\=true,rbac.enabled\=true --namespace gitlab-managed-apps -f /data/helm/runner/config/values.yaml Mounts: /data/helm/runner/config from configuration-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from tiller-token-ddhrt (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: configuration-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: values-content-configuration-runner Optional: false tiller-token-ddhrt: Type: Secret (a volume populated by a Secret) SecretName: tiller-token-ddhrt Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events:
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 12.7.6 Revision: 61654d25b20 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://www.gitlab.158.184.167.118.xip.io HTTP Clone URL: https://www.gitlab.158.184.167.118.xip.io/some-group/some-project.git SSH Clone URL: git@www.gitlab.158.184.167.118.xip.io:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 11.0.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 7/2 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 4 Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished