JupyterHub fails to install with "Something went wrong while installing JupyterHub"
Summary
The JupyterHub application fails to install through the Kubernetes integration with the following error:
Something went wrong while installing JupyterHub
Operation failed. Check pod logs for install-jupyter for more details.
Steps to reproduce
- Create a new project on GitLab.com and navigate to its Operations -> Kubernetes section.
- Create a new Kubernetes cluster through GKE or add an existing one.
- Install Helm Tiller.
- Install Ingress.
- Point a wildcard DNS A record to the provided Ingress IP address and ensure that DNS has propagated the change.
- Set the base domain of the cluster to your chosen domain name.
- Set the Jupyter hostname to your chosen subdomain.
- Install JupyterHub and then observe the error displayed.
Example Project
This was reproduced in this project.
What is the current bug behavior?
Currently, once a user clicks to install JupyterHub through the Kubernetes cluster integration page, the above error is displayed and the installation fails.
What is the expected correct behavior?
We expect JupyterHub to be installed successfully.
Relevant logs and/or screenshots
Output of kubectl logs -n gitlab-managed-apps install-jupyter
+ helm init --upgrade Creating /root/.helm Creating /root/.helm/repository Creating /root/.helm/repository/cache Creating /root/.helm/repository/local Creating /root/.helm/plugins Creating /root/.helm/starters Creating /root/.helm/cache/archive Creating /root/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /root/.helm. Tiller (the Helm server-side component) has been upgraded to the current version. Happy Helming! + seq 1 30 + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (1)... + echo 'Retrying (1)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (2)... + echo 'Retrying (2)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (3)...' + helm version Retrying (3)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (4)... + echo 'Retrying (4)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (5)...' + helm version Retrying (5)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (6)... + echo 'Retrying (6)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (7)... + echo 'Retrying (7)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (8)...' + helm version Retrying (8)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (9)... + echo 'Retrying (9)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (10)...' + helm version Retrying (10)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (11)... + echo 'Retrying (11)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (12)...' + helm version Retrying (12)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (13)...' + helm version Retrying (13)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (14)...' + helm version Retrying (14)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (15)...' + helm version Retrying (15)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (16)...' Retrying (16)... + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (17)... + echo 'Retrying (17)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (18)...' Retrying (18)... + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (19)...' Retrying (19)... + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (20)...' + helm version Retrying (20)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (21)...' Retrying (21)... + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (22)... + echo 'Retrying (22)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (23)...' + helm version Retrying (23)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (24)...' Retrying (24)... + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (25)...' + helm version Retrying (25)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (26)...' + helm version Retrying (26)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (27)...' + helm version Retrying (27)... Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (28)... + echo 'Retrying (28)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s Retrying (29)... + echo 'Retrying (29)...' + helm version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Error: cannot connect to Tiller + sleep 1s + echo 'Retrying (30)...' Retrying (30)... + helm repo add jupyter https://jupyterhub.github.io/helm-chart/ "jupyter" has been added to your repositories + helm repo update Hang tight while we grab the latest from your chart repositories... ...Skip local chart repository ...Successfully got an update from the "jupyter" chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈ Happy Helming!⎈ + helm upgrade jupyter jupyter/jupyterhub --install --reset-values --tls --tls-ca-cert /data/helm/jupyter/config/ca.pem --tls-cert /data/helm/jupyter/config/cert.pem --tls-key /data/helm/jupyter/config/key.pem --version v0.6 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/jupyter/config/values.yaml Release "jupyter" does not exist. Installing it now. Error: timed out waiting for the condition
Output of kubectl describe pod -n gitlab-managed-apps install-jupyter
Name: install-jupyter Namespace: gitlab-managed-apps Priority: 0 PriorityClassName: Node: gke-tristan-jupyter-test-default-pool-f02e3791-0b2c/10.128.15.197 Start Time: Wed, 13 Mar 2019 10:01:10 -0700 Labels: gitlab.org/action=install gitlab.org/application=jupyter Annotations: Status: Failed IP: 10.92.2.5 Containers: helm: Container ID: docker://0fd66424c71cd915e6b1cfe1c7166f4a15c6be35fc65743d3d9e38eb125b5e37 Image: registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7 Image ID: docker-pullable://registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7@sha256:d9d87d9cccef5746a33c7aa6f9d570bddc7479089376bf92809af300aa0f4e28 Port: Host Port: Command: /bin/sh Args: -c $(COMMAND_SCRIPT) State: Terminated Reason: Error Exit Code: 1 Started: Wed, 13 Mar 2019 10:01:18 -0700 Finished: Wed, 13 Mar 2019 10:07:11 -0700 Ready: False Restart Count: 0 Environment: HELM_VERSION: 2.12.3 TILLER_NAMESPACE: gitlab-managed-apps COMMAND_SCRIPT: set -xeo pipefail helm init --upgrade for i in $(seq 1 30); do helm version && break; sleep 1s; echo "Retrying ($i)..."; done helm repo add jupyter https://jupyterhub.github.io/helm-chart/ helm repo update helm upgrade jupyter jupyter/jupyterhub --install --reset-values --tls --tls-ca-cert /data/helm/jupyter/config/ca.pem --tls-cert /data/helm/jupyter/config/cert.pem --tls-key /data/helm/jupyter/config/key.pem --version v0.6 --set rbac.create\=true,rbac.enabled\=true --namespace gitlab-managed-apps -f /data/helm/jupyter/config/values.yaml Mounts: /data/helm/jupyter/config from configuration-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from tiller-token-l8kv2 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: configuration-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: values-content-configuration-jupyter Optional: false tiller-token-l8kv2: Type: Secret (a volume populated by a Secret) SecretName: tiller-token-l8kv2 Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 11m default-scheduler Successfully assigned gitlab-managed-apps/install-jupyter to gke-tristan-jupyter-test-default-pool-f02e3791-0b2c Normal Pulling 11m kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c pulling image "registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7" Normal Pulled 11m kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c Successfully pulled image "registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7" Normal Created 11m kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c Created container Normal Started 11m kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c Started container
Output of checks
This bug happens on GitLab.com, the version at the time of issue creation is 11.9.0-rc3-ee