Dedicate a partition for the local-Path-Provisioner
By default, when longhorn is disabled, on BM deployments with rke2, volumes are created by the local-path-provisioner, that is creating directories into /opt.
Using the new secured image, we have discovered that the logical volume partition for root has only 5 GB as size and also there is no dedicated /opt partition.
Looking on the os-images namespace we have seen below error which triggered our attention:
Image URL changed (new: oci://registry.gitlab.com/sylva-projects/sylva-elements/diskimage-builder/diskimage-builder-hardened:ubuntu-22.04-full, old: ). File ubuntu-22.04-full.qcow2 must be downloaded.
sh: write error: No space left on device
Checking the linux disk space we have seen that there is no space left on the /dev/mapper/vg-lv-root :
dev/mapper/vg-lv_root 4.6G 4.3G 0 100% /
/dev/mapper/vg-lv_home 22G 72K 22G 1% /home
/dev/mapper/vg-lv_tmp 22G 52K 22G 1% /tmp
/dev/mapper/vg-lv_var 22G 1.2G 20G 6% /var
/dev/mapper/vg-lv_kubelet 88G 166M 85G 1% /var/lib/kubelet
/dev/mapper/vg-lv_containerd 132G 27G 101G 21% /var/lib/rancher/rke2/agent/containerd
/dev/mapper/vg-lv_varlog 22G 147M 21G 1% /var/log
/dev/mapper/vg-lv_vartmp 22G 48K 22G 1% /var/tmp
/dev/mapper/vg-lv_etcd 44G 494M 42G 2% /var/lib/rancher/rke2/server/db
/dev/mapper/vg-lv_varlogaudit 22G 4.2M 22G 1% /var/log/audit
While checking on the failing container the mounted partitions, we have seen that the destination is mapped to /opt/local-path-provisioner.
"destination": "/var/www/os-images/",
"type": "bind",
"source": "/opt/local-path-provisioner/pvc-baf9388c-e248-4d9c-98b6-2f512e15c03b_os-images_os-image-server-ubuntu-2204-full-qcow2",
"options": [
"rbind",
"rprivate",
"rw"
]
},
Actually this is the location where all pv claims for local-path-provisioner will be created.
cat /var/lib/kubelet/pods/091ecb63-23b4-46f5-8d8b-234dd8a0a3e5/volumes/kubernetes.io~configmap/config-volume/config.json
{
"nodePathMap": [
{
"node": "DEFAULT_PATH_FOR_NON_LISTED_NODES",
"paths": [
"/opt/local-path-provisioner"
]
}
]
This is the image which was used during this tests:
os_images:
ubuntu-2204-full-qcow2:
uri: "{{ .Values.sylva_base_oci_registry }}/sylva-elements/diskimage-builder/diskimage-builder-hardened:ubuntu-22.04-full"
filename: ubuntu-22.04-full.qcow2
checksum: 6ea1be09d24f09cff3c68356035b0f31d4237c188c75e7562631a02bd2405167