JupyterHub fails to install with "Something went wrong while installing JupyterHub"

Summary

The JupyterHub application fails to install through the Kubernetes integration with the following error:

Something went wrong while installing JupyterHub
Operation failed. Check pod logs for install-jupyter for more details.

Steps to reproduce

  1. Create a new project on GitLab.com and navigate to its Operations -> Kubernetes section.
  2. Create a new Kubernetes cluster through GKE or add an existing one.
  3. Install Helm Tiller.
  4. Install Ingress.
  5. Point a wildcard DNS A record to the provided Ingress IP address and ensure that DNS has propagated the change.
  6. Set the base domain of the cluster to your chosen domain name.
  7. Set the Jupyter hostname to your chosen subdomain.
  8. Install JupyterHub and then observe the error displayed.

Example Project

This was reproduced in this project.

What is the current bug behavior?

Currently, once a user clicks to install JupyterHub through the Kubernetes cluster integration page, the above error is displayed and the installation fails.

What is the expected correct behavior?

We expect JupyterHub to be installed successfully.

Relevant logs and/or screenshots

Output of kubectl logs -n gitlab-managed-apps install-jupyter

+ helm init --upgrade
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!
+ seq 1 30
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (1)...
+ echo 'Retrying (1)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (2)...
+ echo 'Retrying (2)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (3)...'
+ helm version
Retrying (3)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (4)...
+ echo 'Retrying (4)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (5)...'
+ helm version
Retrying (5)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (6)...
+ echo 'Retrying (6)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (7)...
+ echo 'Retrying (7)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (8)...'
+ helm version
Retrying (8)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (9)...
+ echo 'Retrying (9)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (10)...'
+ helm version
Retrying (10)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (11)...
+ echo 'Retrying (11)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (12)...'
+ helm version
Retrying (12)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (13)...'
+ helm version
Retrying (13)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (14)...'
+ helm version
Retrying (14)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (15)...'
+ helm version
Retrying (15)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (16)...'
Retrying (16)...
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (17)...
+ echo 'Retrying (17)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (18)...'
Retrying (18)...
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (19)...'
Retrying (19)...
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (20)...'
+ helm version
Retrying (20)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (21)...'
Retrying (21)...
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (22)...
+ echo 'Retrying (22)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (23)...'
+ helm version
Retrying (23)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (24)...'
Retrying (24)...
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (25)...'
+ helm version
Retrying (25)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (26)...'
+ helm version
Retrying (26)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (27)...'
+ helm version
Retrying (27)...
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (28)...
+ echo 'Retrying (28)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
Retrying (29)...
+ echo 'Retrying (29)...'
+ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Error: cannot connect to Tiller
+ sleep 1s
+ echo 'Retrying (30)...'
Retrying (30)...
+ helm repo add jupyter https://jupyterhub.github.io/helm-chart/
"jupyter" has been added to your repositories
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "jupyter" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈
+ helm upgrade jupyter jupyter/jupyterhub --install --reset-values --tls --tls-ca-cert /data/helm/jupyter/config/ca.pem --tls-cert /data/helm/jupyter/config/cert.pem --tls-key /data/helm/jupyter/config/key.pem --version v0.6 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/jupyter/config/values.yaml
Release "jupyter" does not exist. Installing it now.
Error: timed out waiting for the condition

Output of kubectl describe pod -n gitlab-managed-apps install-jupyter

Name:               install-jupyter
Namespace:          gitlab-managed-apps
Priority:           0
PriorityClassName:  
Node:               gke-tristan-jupyter-test-default-pool-f02e3791-0b2c/10.128.15.197
Start Time:         Wed, 13 Mar 2019 10:01:10 -0700
Labels:             gitlab.org/action=install
                    gitlab.org/application=jupyter
Annotations:        
Status:             Failed
IP:                 10.92.2.5
Containers:
  helm:
    Container ID:  docker://0fd66424c71cd915e6b1cfe1c7166f4a15c6be35fc65743d3d9e38eb125b5e37
    Image:         registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7
    Image ID:      docker-pullable://registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7@sha256:d9d87d9cccef5746a33c7aa6f9d570bddc7479089376bf92809af300aa0f4e28
    Port:          
    Host Port:     
    Command:
      /bin/sh
    Args:
      -c
      $(COMMAND_SCRIPT)
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 13 Mar 2019 10:01:18 -0700
      Finished:     Wed, 13 Mar 2019 10:07:11 -0700
    Ready:          False
    Restart Count:  0
    Environment:
      HELM_VERSION:      2.12.3
      TILLER_NAMESPACE:  gitlab-managed-apps
      COMMAND_SCRIPT:    set -xeo pipefail
helm init --upgrade
for i in $(seq 1 30); do helm version && break; sleep 1s; echo "Retrying ($i)..."; done
helm repo add jupyter https://jupyterhub.github.io/helm-chart/
helm repo update
helm upgrade jupyter jupyter/jupyterhub --install --reset-values --tls --tls-ca-cert /data/helm/jupyter/config/ca.pem --tls-cert /data/helm/jupyter/config/cert.pem --tls-key /data/helm/jupyter/config/key.pem --version v0.6 --set rbac.create\=true,rbac.enabled\=true --namespace gitlab-managed-apps -f /data/helm/jupyter/config/values.yaml
    Mounts:
      /data/helm/jupyter/config from configuration-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from tiller-token-l8kv2 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  configuration-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      values-content-configuration-jupyter
    Optional:  false
  tiller-token-l8kv2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  tiller-token-l8kv2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                          Message
  ----    ------     ----  ----                                                          -------
  Normal  Scheduled  11m   default-scheduler                                             Successfully assigned gitlab-managed-apps/install-jupyter to gke-tristan-jupyter-test-default-pool-f02e3791-0b2c
  Normal  Pulling    11m   kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c  pulling image "registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7"
  Normal  Pulled     11m   kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c  Successfully pulled image "registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.12.3-kube-1.11.7"
  Normal  Created    11m   kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c  Created container
  Normal  Started    11m   kubelet, gke-tristan-jupyter-test-default-pool-f02e3791-0b2c  Started container

Output of checks

This bug happens on GitLab.com, the version at the time of issue creation is 11.9.0-rc3-ee