capm3 upgrade pipeline from 1.1.0 to main is failing on metal3 - longhorn PV not ready for workloads
capm3 upgrade pipeline from 1.1.0 to main failed this night on update mgmt cluster step with the error below :
See: https://gitlab.com/sylva-projects/sylva-core/-/jobs/6689382511
2024/04/22 21:23:55.690331 Kustomization/metal3 state changed: Progressing - Reconciliation in progress
2024/04/22 21:23:56.931257 Command timeout exceeded
Timed-out waiting for the following resources to be ready:
IDENTIFIER STATUS REASON MESSAGE
...
Kustomization/sylva-system/metal3 InProgress Kustomization generation is 1, but latest observed generation is -1
╰┄╴HelmRelease/sylva-system/metal3 InProgress HelmRelease generation is 1, but latest observed generation is -1
├┄╴Deployment/metal3-system/metal3-metal3-ironic InProgress Available: 0/1
┆ ╰┄╴ReplicaSet/metal3-system/metal3-metal3-ironic-7f7d7ffd74 InProgress Available: 0/1
┆ ╰┄╴Pod/metal3-system/metal3-metal3-ironic-7f7d7ffd74-tf6jp InProgress Pod is in the Pending phase
┆ ├┄╴┬┄┄[Conditions]
┆ ┆ ├┄╴Initialized False ContainersNotInitialized containers with incomplete status: [ironic-ipa-downloader]
┆ ┆ ├┄╴Ready False ContainersNotReady containers with unready status: [ironic-httpd ironic-inspector ironic-log-watch ironic]
┆ ┆ ├┄╴ContainersReady False ContainersNotReady containers with unready status: [ironic-httpd ironic-inspector ironic-log-watch ironic]
┆ ┆ ╰┄╴PodScheduled True
┆ ╰┄╴┬┄┄[Events]
┆ ├┄╴metal3-metal3-ironic-7f7d7ffd74-tf6jp.17c8b6d00ed3400b FailedScheduling 0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..
┆ ├┄╴metal3-metal3-ironic-7f7d7ffd74-tf6jp.17c8b6d77b35fe3e FailedScheduling 0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..
┆ ├┄╴metal3-metal3-ironic-7f7d7ffd74-tf6jp.17c8b6d809a5cb29 Scheduled Successfully assigned metal3-system/metal3-metal3-ironic-7f7d7ffd74-tf6jp to mgmt-1263405107-rke2-capm3-virt-management-cp-1
┆ ╰┄╴metal3-metal3-ironic-7f7d7ffd74-tf6jp.17c8b6d82d1e2f2c FailedAttachVolume AttachVolume.Attach failed for volume "pvc-eb5b236b-0ae8-47eb-8878-508033f4dfa2" : rpc error: code = Aborted desc = volume pvc-eb5b236b-0ae8-47eb-8878-508033f4dfa2 is not ready for workloads
╰┄╴Deployment/metal3-system/metal3-metal3-mariadb InProgress Available: 0/1
╰┄╴ReplicaSet/metal3-system/metal3-metal3-mariadb-64947cdbbd InProgress Available: 0/1
╰┄╴Pod/metal3-system/metal3-metal3-mariadb-64947cdbbd-zhjjb InProgress Pod is in the Pending phase
├┄╴┬┄┄[Conditions]
┆ ├┄╴Initialized True
┆ ├┄╴Ready False ContainersNotReady containers with unready status: [mariadb]
┆ ├┄╴ContainersReady False ContainersNotReady containers with unready status: [mariadb]
┆ ╰┄╴PodScheduled True
╰┄╴┬┄┄[Events]
├┄╴metal3-metal3-mariadb-64947cdbbd-zhjjb.17c8b6cffbd5c780 FailedScheduling 0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..
├┄╴metal3-metal3-mariadb-64947cdbbd-zhjjb.17c8b6d809a70c0d Scheduled Successfully assigned metal3-system/metal3-metal3-mariadb-64947cdbbd-zhjjb to mgmt-1263405107-rke2-capm3-virt-management-cp-0
├┄╴metal3-metal3-mariadb-64947cdbbd-zhjjb.17c8b6d77b34f765 FailedScheduling 0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..
╰┄╴metal3-metal3-mariadb-64947cdbbd-zhjjb.17c8b6d82ef68efe FailedAttachVolume AttachVolume.Attach failed for volume "pvc-eb527958-ac6a-434b-83d0-273bc20fb3d8" : rpc error: code = Aborted desc = volume pvc-eb527958-ac6a-434b-83d0-273bc20fb3d8 is not ready for workloads
Edited by Thomas Morin