Unable to install Cilium via GMAv2
Summary
When following the instructions for installing Cilium through GMAv2, the installation fails with a FailedMount
error.
Steps to reproduce
- Create a new Kubernetes cluster in GKE
- Version: 1.20.10-gke.301
- Available nodes: 3
- Node type: e2-standard-4
- Create a new GitLab group
- Within that group create a new project using the Cluster Management Project template
- Go to the GitLab group on the Kubernetes page and connect the cluster you created via the certificate method
- On the
Advanced Settings
tab of the cluster, associate the cluster with the project you created - In your Cluster Management Project, uncomment the
helmfile.yaml
line to install Ingress - In your Kubernetes cluster run
kubectl get services --all-namespaces
to identify the External IP address of the Ingress Loadbalancer - Revisit the group Kubernetes page and update the Base domain field as
<IP_ADDRESS>.nip.io
- Return to the Cluster Management Project and uncomment the line in
helmfile.yaml
to install Cilium - Observe that the CI pipeline fails after about 5 minutes and Cilium is not running in the cluster (see logs below)
Example Project
What is the current bug behavior?
The Cilium pods hang during the Init phase and are never started
What is the expected correct behavior?
Cilium should be installed and start correctly
Relevant logs and/or screenshots
GitLab CI Logs
failed processing release cilium: command "/usr/bin/helm" exited with non-zero status:
Full Log Details
Upgrading release=cilium, chart=cilium/cilium
Release "cilium" does not exist. Installing it now.
FAILED RELEASES:
NAME
cilium
in /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility/helmfile.yaml: in .helmfiles[1]: in applications/cilium/helmfile.yaml: failed processing release cilium: command "/usr/bin/helm" exited with non-zero status:
PATH:
/usr/bin/helm
ARGS:
0: helm (4 bytes)
1: upgrade (7 bytes)
2: --install (9 bytes)
3: --reset-values (14 bytes)
4: cilium (6 bytes)
5: cilium/cilium (13 bytes)
6: --version (9 bytes)
7: 1.10.1 (6 bytes)
8: --wait (6 bytes)
9: --create-namespace (18 bytes)
10: --namespace (11 bytes)
11: gitlab-managed-apps (19 bytes)
12: --values (8 bytes)
13: /tmp/values329253783 (20 bytes)
14: --history-max (13 bytes)
15: 10 (2 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
Error: timed out waiting for the condition
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /builds/dev-sec-product-private/demos/wayne-enterprises/wayne-financial/customer-upload-utility.tmp/KUBECONFIG
Release "cilium" does not exist. Installing it now.
Error: timed out waiting for the condition
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
Kubernetes Cluster Logs
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m11s default-scheduler Successfully assigned gitlab-managed-apps/cilium-552v5 to gke-sec-pm-demo2-default-pool-6c2e31f6-0d9m
Warning FailedMount 8s kubelet Unable to attach or mount volumes: unmounted volumes=[cni-path], unattached volumes=[cilium-cgroup etc-cni-netd clustermesh-secrets hubble-tls cni-path cilium-run lib-modules xtables-lock cilium-token-8kznc bpf-maps cilium-config-path hostproc]: timed out waiting for the condition
Warning FailedMount 3s (x9 over 2m11s) kubelet MountVolume.SetUp failed for volume "cni-path" : mkdir /opt/cni: read-only file system
Full Log Details
swhite@cloudshell:~ (gitlab-demos)$ kubectl get pods -n gitlab-managed-apps
NAME READY STATUS RESTARTS AGE
cilium-552v5 0/2 Init:0/2 0 116s
cilium-7cwmm 0/2 Init:0/2 0 116s
cilium-hj4v9 0/2 Init:0/2 0 116s
cilium-operator-bbccb975f-r7mln 1/1 Running 0 116s
cilium-operator-bbccb975f-ws4qm 1/1 Running 0 116s
hubble-relay-d5bc485c7-d7x62 1/1 Running 0 116s
ingress-nginx-ingress-controller-68bcfdf674-nj6tn 1/1 Running 0 2m6s
ingress-nginx-ingress-default-backend-c9b59c85-856qq 1/1 Running 0 2m6s
swhite@cloudshell:~ (gitlab-demos)$ kubectl describe pod cilium-552v5 -n gitlab-managed-apps
Name: cilium-552v5
Namespace: gitlab-managed-apps
Priority: 2000001000
Priority Class Name: system-node-critical
Node: gke-sec-pm-demo2-default-pool-6c2e31f6-0d9m/10.128.0.118
Start Time: Fri, 29 Oct 2021 21:23:57 +0000
Labels: controller-revision-hash=95fd7846f
k8s-app=cilium
pod-template-generation=1
Annotations: scheduler.alpha.kubernetes.io/critical-pod:
Status: Pending
IP: 10.128.0.118
IPs:
IP: 10.128.0.118
Controlled By: DaemonSet/cilium
Init Containers:
mount-cgroup:
Container ID:
Image: quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
Image ID:
Port: <none>
Host Port: <none>
Command:
sh
-c
cp /usr/bin/cilium-mount /hostbin/cilium-mount && nsenter --cgroup=/hostproc/1/ns/cgroup --mount=/hostproc/1/ns/mnt "${BIN_PATH}/cilium-mount" $CGROUP_ROOT; rm /hostbin/cilium-mount
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
CGROUP_ROOT: /run/cilium/cgroupv2
BIN_PATH: /opt/cni/bin
Mounts:
/hostbin from cni-path (rw)
/hostproc from hostproc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
clean-cilium-state:
Container ID:
Image: quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
Image ID:
Port: <none>
Host Port: <none>
Command:
/init-container.sh
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 100Mi
Environment:
CILIUM_ALL_STATE: <set to the key 'clean-cilium-state' of config map 'cilium-config'> Optional: true
CILIUM_BPF_STATE: <set to the key 'clean-cilium-bpf-state' of config map 'cilium-config'> Optional: true
Mounts:
/run/cilium/cgroupv2 from cilium-cgroup (rw)
/sys/fs/bpf from bpf-maps (rw)
/var/run/cilium from cilium-run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
Containers:
cilium-agent:
Container ID:
Image: quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
Image ID:
Port: 9091/TCP
Host Port: 9091/TCP
Command:
cilium-agent
Args:
--config-dir=/tmp/cilium/config-map
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://127.0.0.1:9876/healthz delay=0s timeout=5s period=30s #success=1 #failure=10
Readiness: http-get http://127.0.0.1:9876/healthz delay=0s timeout=5s period=30s #success=1 #failure=3
Startup: http-get http://127.0.0.1:9876/healthz delay=0s timeout=1s period=2s #success=1 #failure=105
Environment:
K8S_NODE_NAME: (v1:spec.nodeName)
CILIUM_K8S_NAMESPACE: gitlab-managed-apps (v1:metadata.namespace)
CILIUM_CLUSTERMESH_CONFIG: /var/lib/cilium/clustermesh/
CILIUM_CNI_CHAINING_MODE: <set to the key 'cni-chaining-mode' of config map 'cilium-config'> Optional: true
CILIUM_CUSTOM_CNI_CONF: <set to the key 'custom-cni-conf' of config map 'cilium-config'> Optional: true
Mounts:
/host/etc/cni/net.d from etc-cni-netd (rw)
/host/opt/cni/bin from cni-path (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/bpf from bpf-maps (rw)
/tmp/cilium/config-map from cilium-config-path (ro)
/var/lib/cilium/clustermesh from clustermesh-secrets (ro)
/var/lib/cilium/tls/hubble from hubble-tls (ro)
/var/run/cilium from cilium-run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
cilium-monitor:
Container ID:
Image: quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb
Image ID:
Port: <none>
Host Port: <none>
Command:
cilium
Args:
monitor
--type=drop
--type=policy-verdict
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/cilium from cilium-run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from cilium-token-8kznc (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
cilium-run:
Type: HostPath (bare host directory volume)
Path: /var/run/cilium
HostPathType: DirectoryOrCreate
bpf-maps:
Type: HostPath (bare host directory volume)
Path: /sys/fs/bpf
HostPathType: DirectoryOrCreate
hostproc:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType: Directory
cilium-cgroup:
Type: HostPath (bare host directory volume)
Path: /run/cilium/cgroupv2
HostPathType: DirectoryOrCreate
cni-path:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType: DirectoryOrCreate
etc-cni-netd:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType: DirectoryOrCreate
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
clustermesh-secrets:
Type: Secret (a volume populated by a Secret)
SecretName: cilium-clustermesh
Optional: true
cilium-config-path:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: cilium-config
Optional: false
hubble-tls:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: hubble-server-certs
SecretOptionalName: 0xc00084651a
cilium-token-8kznc:
Type: Secret (a volume populated by a Secret)
SecretName: cilium-token-8kznc
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m11s default-scheduler Successfully assigned gitlab-managed-apps/cilium-552v5 to gke-sec-pm-demo2-default-pool-6c2e31f6-0d9m
Warning FailedMount 8s kubelet Unable to attach or mount volumes: unmounted volumes=[cni-path], unattached volumes=[cilium-cgroup etc-cni-netd clustermesh-secrets hubble-tls cni-path cilium-run lib-modules xtables-lock cilium-token-8kznc bpf-maps cilium-config-path hostproc]: timed out waiting for the condition
Warning FailedMount 3s (x9 over 2m11s) kubelet MountVolume.SetUp failed for volume "cni-path" : mkdir /opt/cni: read-only file system
Possible fixes
Edited by Sam White