kubevirt-test-vms remains in InProgress after the KubeVirt upgrade
On upgrade-from-release-1.8, the wkld-upfate-test-units is failing on Ks/kubevirt-test-bms with VirtualMachine/... status: "InProgress" while both of the VMs are Running with Ready=True. kubevirt-test-bms-remove (from !4750 (merged)) is foing the delete of the VMs and they were recreted at generation 1, but this happened while virt-controller was still at 1.7.0. Once the 1.8.3 virt-controller started, it actually added kubevirt.io/pci-topology-version to the VM templates, so generation movead ahead of status.observedGeneration, which KubeVirt only advances on VM restart. With wait: true and no actual healthCheckExprs, Flux is evaluating the VirtualMachine with the generic kstatus rule (generation != observedGeneration => InProgress), soi the unit can not go ready without another restart.
Timeline from the latest observed job (
T1 (12:34 -> 13:15:39) - the workload update rolls all 4 machines (VMs shut down and restarted when md-o is drained at 13:00); the cluster is completley yp to dat at 13:15:39
crustgather-job-17446644712 ~> kubectl -n rke2-capm3-virt get machines -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp,PHASE:.status.phase,NODE:.status.nodeRef.name,UPTODATE:.status.conditions[?(@.type=="UpToDate")].status,DELETING:.metadata.deletionTimestamp'; kubectl -n rke2-capm3-virt get kustomization cluster-machines-ready cluster kubevirt-test-vms -o custom-columns='NAME:.metadata.name,GEN:.metadata.generation,OBS:.status.observedGeneration,READY:.status.conditions[?(@.type=="Ready")].status,SINCE:.status.conditions[?(@.type=="Ready")].lastTransitionTime,REASON:.status.conditions[?(@.type=="Ready")].reason'
NAME CREATED PHASE NODE UPTODATE DELETING
wc-2986208786-rke2-capm3-virt-control-plane-db9nz 2026-10-25T12:34:28Z Running wc-2986208786-rke2-capm3-virt-workload-cp-0 True <none>
wc-2986208786-rke2-capm3-virt-control-plane-lxx4r 2026-10-25T12:43:02Z Running wc-2986208786-rke2-capm3-virt-workload-cp-2 True <none>
wc-2986208786-rke2-capm3-virt-control-plane-pk9v5 2026-10-25T12:54:03Z Running wc-2986208786-rke2-capm3-virt-workload-cp-1 True <none>
wc-2986208786-rke2-capm3-virt-md0-5ptzj-qbsmz 2026-10-25T13:02:16Z Running wc-2986208786-rke2-capm3-virt-workload-md-0 True <none>
NAME GEN OBS READY SINCE REASON
cluster-machines-ready 5 5 True 2026-10-25T13:15:39Z ReconciliationSucceeded
cluster 5 5 True 2026-10-25T13:15:28Z ReconciliationSucceeded
kubevirt-test-vms 5 4 False 2026-10-25T13:24:18Z HealthCheckFailed
crustgather-job-17446644712 ~>T2 (13:08:30 -> 13:09:47) - kubevirt-test-vms-remove runs and the VMs are recreated by Flux. The HelmRelease of Kubevirt is upgraded from 1.7.0 to 1.8.3 at 13:08:38, in the same seconds
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get events -o custom-columns='T:.lastTimestamp,OBJ:.involvedObject.name,R:.reason,MSG:.message' | sort | grep -vE 'Pulled|Pulling|virt-launcher' | cut -c1-180 | head -20
<nil> kubevirt-test-vms-remove-sf2tr Scheduled Successfully assigned kubevirt-tests/kubevirt-test-vms-remove-sf2tr to wc-2986208786-rke2-capm3-v
2026-10-25T10:51:06Z cirros-kubevirt-vm SuccessfulCreate Started the virtual machine by creating the new virtual machine instance cirros-kubevirt-vm
2026-10-25T10:51:06Z multus-kubevirt-vm SuccessfulCreate Started the virtual machine by creating the new virtual machine instance multus-kubevirt-vm
2026-10-25T10:51:28Z cirros-kubevirt-vm Created VirtualMachineInstance defined.
2026-10-25T10:51:28Z cirros-kubevirt-vm Started VirtualMachineInstance started.
2026-10-25T10:51:28Z multus-kubevirt-vm Created VirtualMachineInstance defined.
2026-10-25T10:51:28Z multus-kubevirt-vm Started VirtualMachineInstance started.
2026-10-25T13:00:30Z cirros-kubevirt-vm ShuttingDown Signaled Graceful Shutdown
2026-10-25T13:00:30Z multus-kubevirt-vm ShuttingDown Signaled Graceful Shutdown
2026-10-25T13:00:33Z cirros-kubevirt-vm Stopped The VirtualMachineInstance was shut down.
2026-10-25T13:00:33Z multus-kubevirt-vm Deleted Signaled Deletion
2026-10-25T13:00:33Z multus-kubevirt-vm Stopped The VirtualMachineInstance was shut down.
2026-10-25T13:00:33Z multus-kubevirt-vm SuccessfulDelete Stopped the virtual machine by deleting the virtual machine instance cf51d2b2-3f89-4e21-b795-aa97
2026-10-25T13:00:34Z cirros-kubevirt-vm Deleted Signaled Deletion
2026-10-25T13:00:34Z cirros-kubevirt-vm SuccessfulDelete Stopped the virtual machine by deleting the virtual machine instance 3f3ff710-5253-4ef9-b573-4aab
2026-10-25T13:00:39Z cirros-kubevirt-vm SuccessfulCreate Started the virtual machine by creating the new virtual machine instance cirros-kubevirt-vm
2026-10-25T13:00:40Z multus-kubevirt-vm SuccessfulCreate Started the virtual machine by creating the new virtual machine instance multus-kubevirt-vm
2026-10-25T13:08:30Z kubevirt-test-vms-remove SuccessfulCreate Created pod: kubevirt-test-vms-remove-sf2tr
2026-10-25T13:08:31Z kubevirt-test-vms-remove-sf2tr AddedInterface Add eth0 [100.72.157.238/32] from k8s-pod-network
2026-10-25T13:08:37Z kubevirt-test-vms-remove-sf2tr Created Container created
crustgather-job-17446644712 ~> kubectl -n rke2-capm3-virt get helmrelease kubevirt -o jsonpath='{range .status.history[*]}{.version} {.chartVersion} {.appVersion} {.lastDeployed}{"\n"}{end}'
2 0.8.0 1.8.3 2026-10-25T13:08:38Z
1 0.7.0 1.7.0 2026-10-25T10:48:23Z
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get vm -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp,GEN:.metadata.generation,OBS:.status.observedGeneration'
NAME CREATED GEN OBS
cirros-kubevirt-vm 2026-10-25T13:09:47Z 3 1
multus-kubevirt-vm 2026-10-25T13:09:47Z 3 1
crustgather-job-17446644712 ~>T3 (13:09:55) - the recreated VMs are Running and Ready. Their VMI is at VM generation 1 (Consequence of T2)
crustgather-job-17446644712 ~> kubectl config use-context workload && kubectl -n kubevirt-tests get vm,vmi,pods -o wide
Switched to context "workload".
NAME STATUS READY AGE
cirros-kubevirt-vm Running True 19h
multus-kubevirt-vm Running True 19h
NAME PHASE IP NODENAME READY LIVE-MIGRATABLE PAUSED AGE
cirros-kubevirt-vm Running 100.72.157.242 wc-2986208786-rke2-capm3-virt-workload-md-0 True True 19h
multus-kubevirt-vm Running 100.72.157.243 wc-2986208786-rke2-capm3-virt-workload-md-0 True True 19h
NAME READY STATUS RESTARTS AGE
virt-launcher-cirros-kubevirt-vm-2tvck 2/2 Running 0 19h
virt-launcher-multus-kubevirt-vm-79sld 2/2 Running 0 19h
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get vm cirros-kubevirt-vm -o jsonpath='{range .status.conditions[*]}{.lastTransitionTime} {.type}={.status} {.reason}: {.message}{"\n"}{end}'; kubectl --context workload -n kubevirt-tests get vmi cirros-kubevirt-vm -o jsonpath='{.metadata.annotations.kubevirt\.io/vm-generation}{"\n"}'
2026-10-25T13:09:55Z Ready=True :
null LiveMigratable=True :
null StorageLiveMigratable=True :
1
crustgather-job-17446644712 ~>T4 (13:13:22 -> 13:14:23) - the 1.8.3 virt-controller pods are starting and are writing kubevirt.io/pci-topology-version into the template of the existing VMs. Kustomize-controller is re-applying at 13:15:48. The VMs end up at generation 3 with observedGeneration: 1 (Consequence of T2 and T3)
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt get pods -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp,STARTED:.status.containerStatuses[0].state.running.startedAt,IMAGE:.spec.containers[0].image' | grep -E 'NAME|virt-controller|virt-operator'
NAME CREATED STARTED IMAGE
virt-controller-67f5779f8f-bvmzm 2026-10-25T13:13:16Z 2026-10-25T13:13:22Z registry.suse.com/suse/sles/16.0/virt-controller:1.8.3
virt-controller-67f5779f8f-fzl8b 2026-10-25T13:13:42Z 2026-10-25T13:13:47Z registry.suse.com/suse/sles/16.0/virt-controller:1.8.3
virt-operator-54bc4df697-rwgvb 2026-10-25T13:08:52Z 2026-10-25T13:09:09Z registry.suse.com/suse/sles/16.0/virt-operator:1.8.3
virt-operator-54bc4df697-w78nx 2026-10-25T13:09:20Z 2026-10-25T13:09:27Z registry.suse.com/suse/sles/16.0/virt-operator:1.8.3
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get vm cirros-kubevirt-vm -o jsonpath='{range .metadata.managedFields[*]}== {.manager} {.operation} {.subresource} {.time}{"\n"}{.fieldsV1}{"\n"}{end}' | cut -c1-700
== kustomize-controller Apply 2026-10-25T13:15:48Z
{"f:metadata":{"f:labels":{"f:kustomize.toolkit.fluxcd.io/name":{},"f:kustomize.toolkit.fluxcd.io/namespace":{},"f:sylva.org/sylva-managed":{}}},"f:spec":{"f:runStrategy":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:kubevirt.io/domain":{},"f:kubevirt.io/size":{}}},"f:spec":{"f:affinity":{"f:nodeAffinity":{"f:requiredDuringSchedulingIgnoredDuringExecution":{}}},"f:domain":{"f:cpu":{"f:cores":{}},"f:devices":{"f:disks":{},"f:interfaces":{}},"f:resources":{"f:requests":{"f:memory":{}}}},"f:networks":{},"f:volumes":{}}}}}
== virt-controller Update 2026-10-25T13:14:23Z
{"f:metadata":{"f:annotations":{".":{},"f:kubevirt.io/latest-observed-api-version":{},"f:kubevirt.io/storage-observed-api-version":{}},"f:finalizers":{".":{},"v:\"kubevirt.io/virtualMachineControllerFinalize\"":{}}},"f:spec":{"f:template":{"f:metadata":{"f:annotations":{".":{},"f:kubevirt.io/pci-topology-version":{}}}}}}
== virt-controller Update status 2026-10-25T13:15:48Z
{"f:status":{".":{},"f:conditions":{},"f:created":{},"f:desiredGeneration":{},"f:observedGeneration":{},"f:printableStatus":{},"f:ready":{},"f:runStrategy":{},"f:volumeSnapshotStatuses":{}}}
crustgather-job-17446644712 ~>Also the manifest of the unit is identical between 1.8 and main
[git:main]root@tibi-vbmh:sylva-core# git pull
remote: Enumerating objects: 163, done.
remote: Counting objects: 100% (103/103), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 47 (delta 32), reused 35 (delta 22), pack-reused 0 (from 0)
Unpacking objects: 100% (47/47), 9.98 KiB | 150.00 KiB/s, done.
From https://gitlab.com/sylva-projects/sylva-core
68ef2e1bf..48d9fe9a8 ba/test-branch -> origin/ba/test-branch
* [new branch] cherry-pick-b1b73d16 -> origin/cherry-pick-b1b73d16
* [new branch] dg/kyverno-cel-policies-part2 -> origin/dg/kyverno-cel-policies-part2
da0dde49e..79079d742 mohan/create-xfs-sc-for-nfs-ganesha -> origin/mohan/create-xfs-sc-for-nfs-ganesha
Already up to date.
[git:main]root@tibi-vbmh:sylva-core# git diff origin/release-1.8 origin/main --stat -- kustomize-units/kubevirt-test-vms/
[git:main]root@tibi-vbmh:sylva-core#T5 (13:24:18 -> 13:43:50) - kubevirt-test-vms fails its healthcheck with both VMs InProgress and wkld-update-test-units times out after 10m (consequence of T4)
crustgather-job-17446644712 ~> kubectl --context management -n rke2-capm3-virt get kustomization kubevirt-test-vms -o jsonpath='{range .status.conditions[*]}{.lastTransitionTime} {.type}={.status} {.reason}: {.message}{"\n"}{end}'
2026-10-25T13:24:18Z Reconciling=True ProgressingWithRetry: Running health checks for revision 0.0.0-git-4352f62c@sha256:838e76a51ec74872687ebec7ed056febad0f04de7a5f48e9e155685fa5a6601c with a timeout of 30s
2026-10-25T13:24:18Z Ready=False HealthCheckFailed: health check failed after 30.033155722s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026-10-25T13:24:18Z Healthy=False HealthCheckFailed: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
crustgather-job-17446644712 ~>From the job log of https://gitlab.com/sylva-projects/sylva-core/-/jobs/17446644731
2026/10/25 13:40:12.475075 Kustomization/kubevirt-test-vms state changed: HealthCheckFailed - health check failed after 30.029382528s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026/10/25 13:41:12.553761 Kustomization/kubevirt-test-vms state changed: Progressing - Reconciliation in progress
2026/10/25 13:41:42.988387 Kustomization/kubevirt-test-vms state changed: HealthCheckFailed - health check failed after 30.032704132s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026/10/25 13:42:43.094293 Kustomization/kubevirt-test-vms state changed: Progressing - Reconciliation in progress
2026/10/25 13:43:13.607384 Kustomization/kubevirt-test-vms state changed: HealthCheckFailed - health check failed after 30.03270828s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026/10/25 13:43:50.045678 Command timeout exceeded: Kustomization/rke2-capm3-virt/sylva-units-tests-status did not become Ready after 10m0sThe unit uses wait: true with no healthCheckExprs :
[git:main]root@tibi-vbmh:sylva-core# git grep -n -A25 '^ kubevirt-test-vms:' origin/main -- charts/sylva-units/values.yaml | grep -E 'wait|healthCheck|path:'
origin/main:charts/sylva-units/values.yaml-1354- path: ./kustomize-units/kubevirt-test-vms
origin/main:charts/sylva-units/values.yaml-1355- wait: true
[git:main]root@tibi-vbmh:sylva-core#Why kubevirt-test-vms-remove is not enough here:
It actually recreates the VMs while the HelmRelease of KubeVirt is being upgraded (13:08:30 vs 13:08:38), but the new virt-controller only becomes active ~ 5 minutes later (13:13:22) and then is modifying the freshly recreated VMs. ANy restart/recreate done before the new virt-controller is active is undone by it and ordering the remove unit after kubevirt would not reliable either, since the HelmRelease is Ready before virt-operator has finished rolling the components