Skip to content

Unable to install Cilium via GMAv2

Summary

When following the instructions for installing Cilium through GMAv2, the installation fails with a FailedMount error.

Steps to reproduce

  1. Create a new Kubernetes cluster in GKE
    1. Version: 1.20.10-gke.301
    2. Available nodes: 3
    3. Node type: e2-standard-4
  2. Create a new GitLab group
  3. Within that group create a new project using the Cluster Management Project template
  4. Go to the GitLab group on the Kubernetes page and connect the cluster you created via the certificate method
  5. On the Advanced Settings tab of the cluster, associate the cluster with the project you created
  6. In your Cluster Management Project, uncomment the helmfile.yaml line to install Ingress
  7. In your Kubernetes cluster run kubectl get services --all-namespaces to identify the External IP address of the Ingress Loadbalancer
  8. Revisit the group Kubernetes page and update the Base domain field as <IP_ADDRESS>.nip.io
  9. Return to the Cluster Management Project and uncomment the line in helmfile.yaml to install Cilium
  10. Observe that the CI pipeline fails after about 5 minutes and Cilium is not running in the cluster (see logs below)

Example Project

What is the current bug behavior?

The Cilium pods hang during the Init phase and are never started

What is the expected correct behavior?

Cilium should be installed and start correctly

Relevant logs and/or screenshots

GitLab CI Logs

failed processing release cilium: command "/usr/bin/helm" exited with non-zero status:
Full Log Details 
Upgrading release=cilium, chart=cilium/cilium
Release "cilium" does not exist. Installing it now.
FAILED RELEASES:
NAME
cilium
in /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility/helmfile.yaml: in .helmfiles[1]: in applications/cilium/helmfile.yaml: failed processing release cilium: command "/usr/bin/helm" exited with non-zero status:
PATH:
  /usr/bin/helm
ARGS:
  0: helm (4 bytes)
  1: upgrade (7 bytes)
  2: --install (9 bytes)
  3: --reset-values (14 bytes)
  4: cilium (6 bytes)
  5: cilium/cilium (13 bytes)
  6: --version (9 bytes)
  7: 1.10.1 (6 bytes)
  8: --wait (6 bytes)
  9: --create-namespace (18 bytes)
  10: --namespace (11 bytes)
  11: gitlab-managed-apps (19 bytes)
  12: --values (8 bytes)
  13: /tmp/values329253783 (20 bytes)
  14: --history-max (13 bytes)
  15: 10 (2 bytes)
ERROR:
  exit status 1
EXIT STATUS
  1
STDERR:
  WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
  WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
  Error: timed out waiting for the condition
COMBINED OUTPUT:
  WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
  WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
  Release "cilium" does not exist. Installing it now.
  Error: timed out waiting for the condition
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Kubernetes Cluster Logs

Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    2m11s               default-scheduler  Successfully assigned gitlab-managed-apps/cilium-552v5 to gke-sec-pm-demo2-default-pool-6c2e31f6-0d9m
  Warning  FailedMount  8s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cni-path], unattached volumes=[cilium-cgroup etc-cni-netd clustermesh-secrets hubble-tls cni-path cilium-run lib-modules xtables-lock cilium-token-8kznc bpf-maps cilium-config-path hostproc]: timed out waiting for the condition
  Warning  FailedMount  3s (x9 over 2m11s)  kubelet            MountVolume.SetUp failed for volume "cni-path" : mkdir /opt/cni: read-only file system
Full Log Details 
swhite@cloudshell:~ (gitlab-demos)$ kubectl get pods -n gitlab-managed-apps
NAME                                                   READY   STATUS     RESTARTS   AGE
cilium-552v5                                           0/2     Init:0/2   0          116s
cilium-7cwmm                                           0/2     Init:0/2   0          116s
cilium-hj4v9                                           0/2     Init:0/2   0          116s
cilium-operator-bbccb975f-r7mln                        1/1     Running    0          116s
cilium-operator-bbccb975f-ws4qm                        1/1     Running    0          116s
hubble-relay-d5bc485c7-d7x62                           1/1     Running    0          116s
ingress-nginx-ingress-controller-68bcfdf674-nj6tn      1/1     Running    0          2m6s
ingress-nginx-ingress-default-backend-c9b59c85-856qq   1/1     Running    0          2m6s
swhite@cloudshell:~ (gitlab-demos)$ kubectl describe pod cilium-552v5 -n gitlab-managed-apps
Name:                 cilium-552v5
Namespace:            gitlab-managed-apps
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 gke-sec-pm-demo2-default-pool-6c2e31f6-0d9m/10.128.0.118
Start Time:           Fri, 29 Oct 2021 21:23:57 +0000
Labels:               controller-revision-hash=95fd7846f
                      k8s-app=cilium
                      pod-template-generation=1
Annotations:          scheduler.alpha.kubernetes.io/critical-pod:
Status:               Pending
IP:                   10.128.0.118
IPs:
  IP:           10.128.0.118
Controlled By:  DaemonSet/cilium
Init Containers:
  mount-cgroup:
    Container ID:
    Image:         quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      cp /usr/bin/cilium-mount /hostbin/cilium-mount && nsenter --cgroup=/hostproc/1/ns/cgroup --mount=/hostproc/1/ns/mnt "${BIN_PATH}/cilium-mount" $CGROUP_ROOT; rm /hostbin/cilium-mount
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      CGROUP_ROOT:  /run/cilium/cgroupv2
      BIN_PATH:     /opt/cni/bin
    Mounts:
      /hostbin from cni-path (rw)
      /hostproc from hostproc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
  clean-cilium-state:
    Container ID:
    Image:         quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /init-container.sh
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      CILIUM_ALL_STATE:  <set to the key 'clean-cilium-state' of config map 'cilium-config'>      Optional: true
      CILIUM_BPF_STATE:  <set to the key 'clean-cilium-bpf-state' of config map 'cilium-config'>  Optional: true
    Mounts:
      /run/cilium/cgroupv2 from cilium-cgroup (rw)
      /sys/fs/bpf from bpf-maps (rw)
      /var/run/cilium from cilium-run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
Containers:
  cilium-agent:
    Container ID:
    Image:         quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
    Image ID:
    Port:          9091/TCP
    Host Port:     9091/TCP
    Command:
      cilium-agent
    Args:
      --config-dir=/tmp/cilium/config-map
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://127.0.0.1:9876/healthz delay=0s timeout=5s period=30s #success=1 #failure=10
    Readiness:      http-get http://127.0.0.1:9876/healthz delay=0s timeout=5s period=30s #success=1 #failure=3
    Startup:        http-get http://127.0.0.1:9876/healthz delay=0s timeout=1s period=2s #success=1 #failure=105
    Environment:
      K8S_NODE_NAME:               (v1:spec.nodeName)
      CILIUM_K8S_NAMESPACE:       gitlab-managed-apps (v1:metadata.namespace)
      CILIUM_CLUSTERMESH_CONFIG:  /var/lib/cilium/clustermesh/
      CILIUM_CNI_CHAINING_MODE:   <set to the key 'cni-chaining-mode' of config map 'cilium-config'>  Optional: true
      CILIUM_CUSTOM_CNI_CONF:     <set to the key 'custom-cni-conf' of config map 'cilium-config'>    Optional: true
    Mounts:
      /host/etc/cni/net.d from etc-cni-netd (rw)
      /host/opt/cni/bin from cni-path (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/bpf from bpf-maps (rw)
      /tmp/cilium/config-map from cilium-config-path (ro)
      /var/lib/cilium/clustermesh from clustermesh-secrets (ro)
      /var/lib/cilium/tls/hubble from hubble-tls (ro)
      /var/run/cilium from cilium-run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
  cilium-monitor:
    Container ID:
    Image:         quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      cilium
    Args:
      monitor
      --type=drop
      --type=policy-verdict
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/cilium from cilium-run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  cilium-run:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/cilium
    HostPathType:  DirectoryOrCreate
  bpf-maps:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/bpf
    HostPathType:  DirectoryOrCreate
  hostproc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  Directory
  cilium-cgroup:
    Type:          HostPath (bare host directory volume)
    Path:          /run/cilium/cgroupv2
    HostPathType:  DirectoryOrCreate
  cni-path:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  DirectoryOrCreate
  etc-cni-netd:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  DirectoryOrCreate
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  clustermesh-secrets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-clustermesh
    Optional:    true
  cilium-config-path:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      cilium-config
    Optional:  false
  hubble-tls:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          hubble-server-certs
    SecretOptionalName:  0xc00084651a
  cilium-token-8kznc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-token-8kznc
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    2m11s               default-scheduler  Successfully assigned gitlab-managed-apps/cilium-552v5 to gke-sec-pm-demo2-default-pool-6c2e31f6-0d9m
  Warning  FailedMount  8s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cni-path], unattached volumes=[cilium-cgroup etc-cni-netd clustermesh-secrets hubble-tls cni-path cilium-run lib-modules xtables-lock cilium-token-8kznc bpf-maps cilium-config-path hostproc]: timed out waiting for the condition
  Warning  FailedMount  3s (x9 over 2m11s)  kubelet            MountVolume.SetUp failed for volume "cni-path" : mkdir /opt/cni: read-only file system

Possible fixes

Edited by Sam White