Cluster-bmh mistakenly considered failed after pivot
On a local virtual baremetal deployment my sushy emulator flapped during the cluster bootstrap. After fixing it, the deployment continued fine, but after pivot and the reinstantiation of cluster-bmh in the management cluster failed with:
cluster-bmh 27m False Helm upgrade failed for release sylva-system/cluster-bmh with chart sylva-capi-cluster@0.0.0+7db335efa7f1: failed early due to stalled resources: [BareMetalHost/sylva-system/management-cluster-server1 status: 'Failed']After talking with @tmmorin we got to the conclusion that this is caused by the failed criteria of the new healthCheckExprs because the count was > 0. Unfortunately I didn't manage to reproduce the error (still trying), but we believe removing the failure criteria would not do any harm for this type of situations that may appear in prod also.
later edit: I managed to force reproduce it, here is the bmh definition after pivot:
[git:main]root@home-server:sylva-core# kubectl get bmh management-cluster-server1 -o yaml
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
annotations:
interface-mac.sylva.org/ens3: "52:54:01:01:00:01"
interface-mac.sylva.org/ens4: "52:54:01:02:00:01"
meta.helm.sh/release-name: cluster-bmh
meta.helm.sh/release-namespace: sylva-system
sylvaproject.org/baremetal-host-name: server1
sylvaproject.org/cluster-name: management-cluster
sylvaproject.org/default-longhorn-disks-config: W3siYWxsb3dTY2hlZHVsaW5nIjp0cnVlLCJwYXRoIjoiL3Zhci9sb25naG9ybi9kaXNrcy9kaXNrX2J5LXBhdGhfcGNpLTAwMDA6MDA6MDYuMC1zY3NpLTA6MDowOjEiLCJzdG9yYWdlUmVzZXJ2ZWQiOjAsInRhZ3MiOlsidmhkZCIsImZhc3QiXX1d
sylvaproject.org/snmp-auth: dell1
sylvaproject.org/snmp-endpoint: http://192.168.130.11:5432
sylvaproject.org/snmp-hw-type: dell_idrac
creationTimestamp: "2026-06-09T16:05:01Z"
finalizers:
- baremetalhost.metal3.io
generation: 4
labels:
app.kubernetes.io/instance: cluster-bmh
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: sylva-capi-cluster
app.kubernetes.io/version: 0.0.0
cluster-role: control-plane
cluster.x-k8s.io/cluster-name: management-cluster
helm.sh/chart: sylva-capi-cluster-0.0.0_7db335efa7f1
helm.toolkit.fluxcd.io/name: cluster-bmh
helm.toolkit.fluxcd.io/namespace: sylva-system
host-type: generic
snmp-enabled: "true"
sylva.org/sylva-managed: "true"
name: management-cluster-server1
namespace: sylva-system
ownerReferences:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
controller: true
kind: Metal3Machine
name: management-cluster-cp-7a4dd7c452-bmpj5
uid: 0c563fef-9cb9-4957-af1f-843866594937
resourceVersion: "60418"
uid: 3f55f8c0-077e-4e03-b0b1-b48fdd3a72d1
spec:
architecture: x86_64
automatedCleaningMode: disabled
bmc:
address: redfish-virtualmedia://192.168.133.1:5000/redfish/v1/Systems/mgmt-1
credentialsName: management-cluster-server1-secret
disableCertificateVerification: true
bootMACAddress: "52:54:01:02:00:01"
bootMode: legacy
consumerRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3Machine
name: management-cluster-cp-7a4dd7c452-bmpj5
namespace: sylva-system
description: mgmt-control-plane
externallyProvisioned: false
image:
checksum: b432500fd052a98e80b49c0deb8d30ccaaa1577045b13e8231eb23b00fe8d98a
checksumType: sha256
format: raw
url: https://192.168.130.1/ubuntu-noble-hardened-rke2-1-35-0.11.5.raw
metaData:
name: management-cluster-cp-7a4dd7c452-bmpj5-metadata
namespace: sylva-system
networkData:
name: management-cluster-cp-7a4dd7c452-bmpj5-networkdata
namespace: sylva-system
online: true
preprovisioningNetworkDataName: server1-preprov-netdata-05d01bb5
rootDeviceHints:
deviceName: /dev/sda
userData:
name: management-cluster-control-plane-nztvg
namespace: sylva-system
status:
errorCount: 3
errorMessage: ""
goodCredentials:
credentials:
name: management-cluster-server1-secret
namespace: sylva-system
credentialsVersion: "54732"
hardware:
cpu:
arch: x86_64
count: 8
flags:
- 3dnowprefetch
- abm
- adx
- aes
- apic
- arat
- arch_capabilities
- arch_perfmon
- avx
- avx2
- bmi1
- bmi2
- clflush
- cmov
- constant_tsc
- cpuid
- cpuid_fault
- cx16
- cx8
- de
- ept
- ept_ad
- erms
- f16c
- flexpriority
- flush_l1d
- fma
- fpu
- fsgsbase
- fxsr
- hle
- hypervisor
- ibpb
- ibrs
- invpcid
- lahf_lm
- lm
- mca
- mce
- md_clear
- mmx
- movbe
- msr
- mtrr
- nopl
- nx
- pae
- pat
- pcid
- pclmulqdq
- pdcm
- pdpe1gb
- pge
- pni
- popcnt
- pse
- pse36
- pti
- rdrand
- rdseed
- rdtscp
- rep_good
- rtm
- sep
- smap
- smep
- ss
- ssbd
- sse
- sse2
- sse4_1
- sse4_2
- ssse3
- stibp
- syscall
- tpr_shadow
- tsc
- tsc_adjust
- tsc_deadline_timer
- tsc_known_freq
- umip
- vme
- vmx
- vnmi
- vpid
- x2apic
- xsave
- xsaveopt
- xtopology
model: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
firmware:
bios:
date: 04/01/2014
vendor: SeaBIOS
version: 1.17.0-debian-1.17.0-1ubuntu1
hostname: localhost.localdomain
nics:
- mac: "52:54:01:01:00:01"
model: 0x1af4 0x0001
name: ens3
- ip: 192.168.130.11
mac: "52:54:01:02:00:01"
model: 0x1af4 0x0001
name: ens4
ramMebibytes: 16000
storage:
- alternateNames:
- /dev/sda
- /dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:0
hctl: "0:0:0:0"
model: QEMU HARDDISK
name: /dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:0
rotational: true
serialNumber: drive-scsi0-0-0-0
sizeBytes: 107374182400
type: HDD
vendor: QEMU
- alternateNames:
- /dev/sdb
- /dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:1
hctl: "0:0:0:1"
model: QEMU HARDDISK
name: /dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:1
rotational: true
serialNumber: drive-scsi0-0-0-1
sizeBytes: 161061273600
type: HDD
vendor: QEMU
systemVendor:
manufacturer: QEMU
productName: Ubuntu 26.04 PC (i440FX + PIIX, 1996)
hardwareProfile: unknown
lastUpdated: "2026-06-09T16:07:43Z"
operationHistory:
deprovision:
end: null
start: null
inspect:
end: "2026-06-09T14:35:36Z"
start: "2026-06-09T14:32:36Z"
provision:
end: "2026-06-09T15:26:30Z"
start: "2026-06-09T15:13:32Z"
register:
end: "2026-06-09T16:07:43Z"
start: "2026-06-09T16:05:13Z"
operationalStatus: OK
poweredOn: true
provisioning:
ID: 5a54f4ec-146a-4216-b195-07afa22bf085
bootMode: legacy
image:
checksum: b432500fd052a98e80b49c0deb8d30ccaaa1577045b13e8231eb23b00fe8d98a
checksumType: sha256
format: raw
url: https://192.168.130.1/ubuntu-noble-hardened-rke2-1-35-0.11.5.raw
rootDeviceHints:
deviceName: /dev/sda
state: provisioned
triedCredentials:
credentials:
name: management-cluster-server1-secret
namespace: sylva-system
credentialsVersion: "54732"cc @tmmorin ^^
Edited by Cristian Manda