kubevirt-test-vms remains in InProgress after the KubeVirt upgrade

On upgrade-from-release-1.8, the wkld-upfate-test-units is failing on Ks/kubevirt-test-bms with VirtualMachine/... status: "InProgress" while both of the VMs are Running with Ready=True. kubevirt-test-bms-remove (from !4750 (merged)) is foing the delete of the VMs and they were recreted at generation 1, but this happened while virt-controller was still at 1.7.0. Once the 1.8.3 virt-controller started, it actually added kubevirt.io/pci-topology-version to the VM templates, so generation movead ahead of status.observedGeneration, which KubeVirt only advances on VM restart. With wait: true and no actual healthCheckExprs, Flux is evaluating the VirtualMachine with the generic kstatus rule (generation != observedGeneration => InProgress), soi the unit can not go ready without another restart.

Timeline from the latest observed job (🚀 ☁️capm3 🚀rke2 🏗️virt-leaseweb 🎬upgrade-from-release-1.8 🛠️ha,misc ❌ wkld-update-test-units 26.10.2026)

T1 (12:34 -> 13:15:39) - the workload update rolls all 4 machines (VMs shut down and restarted when md-o is drained at 13:00); the cluster is completley yp to dat at 13:15:39

crustgather-job-17446644712 ~> kubectl -n rke2-capm3-virt get machines -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp,PHASE:.status.phase,NODE:.status.nodeRef.name,UPTODATE:.status.conditions[?(@.type=="UpToDate")].status,DELETING:.metadata.deletionTimestamp'; kubectl -n rke2-capm3-virt get kustomization cluster-machines-ready cluster kubevirt-test-vms -o custom-columns='NAME:.metadata.name,GEN:.metadata.generation,OBS:.status.observedGeneration,READY:.status.conditions[?(@.type=="Ready")].status,SINCE:.status.conditions[?(@.type=="Ready")].lastTransitionTime,REASON:.status.conditions[?(@.type=="Ready")].reason'
NAME                                                CREATED                PHASE     NODE                                          UPTODATE   DELETING
wc-2986208786-rke2-capm3-virt-control-plane-db9nz   2026-10-25T12:34:28Z   Running   wc-2986208786-rke2-capm3-virt-workload-cp-0   True       <none>
wc-2986208786-rke2-capm3-virt-control-plane-lxx4r   2026-10-25T12:43:02Z   Running   wc-2986208786-rke2-capm3-virt-workload-cp-2   True       <none>
wc-2986208786-rke2-capm3-virt-control-plane-pk9v5   2026-10-25T12:54:03Z   Running   wc-2986208786-rke2-capm3-virt-workload-cp-1   True       <none>
wc-2986208786-rke2-capm3-virt-md0-5ptzj-qbsmz       2026-10-25T13:02:16Z   Running   wc-2986208786-rke2-capm3-virt-workload-md-0   True       <none>
NAME                     GEN   OBS   READY   SINCE                  REASON
cluster-machines-ready   5     5     True    2026-10-25T13:15:39Z   ReconciliationSucceeded
cluster                  5     5     True    2026-10-25T13:15:28Z   ReconciliationSucceeded
kubevirt-test-vms        5     4     False   2026-10-25T13:24:18Z   HealthCheckFailed
crustgather-job-17446644712 ~>

T2 (13:08:30 -> 13:09:47) - kubevirt-test-vms-remove runs and the VMs are recreated by Flux. The HelmRelease of Kubevirt is upgraded from 1.7.0 to 1.8.3 at 13:08:38, in the same seconds

crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get events -o custom-columns='T:.lastTimestamp,OBJ:.involvedObject.name,R:.reason,MSG:.message' | sort | grep -vE 'Pulled|Pulling|virt-launcher' | cut -c1-180 | head -20
<nil>                  kubevirt-test-vms-remove-sf2tr           Scheduled          Successfully assigned kubevirt-tests/kubevirt-test-vms-remove-sf2tr to wc-2986208786-rke2-capm3-v
2026-10-25T10:51:06Z   cirros-kubevirt-vm                       SuccessfulCreate   Started the virtual machine by creating the new virtual machine instance cirros-kubevirt-vm
2026-10-25T10:51:06Z   multus-kubevirt-vm                       SuccessfulCreate   Started the virtual machine by creating the new virtual machine instance multus-kubevirt-vm
2026-10-25T10:51:28Z   cirros-kubevirt-vm                       Created            VirtualMachineInstance defined.
2026-10-25T10:51:28Z   cirros-kubevirt-vm                       Started            VirtualMachineInstance started.
2026-10-25T10:51:28Z   multus-kubevirt-vm                       Created            VirtualMachineInstance defined.
2026-10-25T10:51:28Z   multus-kubevirt-vm                       Started            VirtualMachineInstance started.
2026-10-25T13:00:30Z   cirros-kubevirt-vm                       ShuttingDown       Signaled Graceful Shutdown
2026-10-25T13:00:30Z   multus-kubevirt-vm                       ShuttingDown       Signaled Graceful Shutdown
2026-10-25T13:00:33Z   cirros-kubevirt-vm                       Stopped            The VirtualMachineInstance was shut down.
2026-10-25T13:00:33Z   multus-kubevirt-vm                       Deleted            Signaled Deletion
2026-10-25T13:00:33Z   multus-kubevirt-vm                       Stopped            The VirtualMachineInstance was shut down.
2026-10-25T13:00:33Z   multus-kubevirt-vm                       SuccessfulDelete   Stopped the virtual machine by deleting the virtual machine instance cf51d2b2-3f89-4e21-b795-aa97
2026-10-25T13:00:34Z   cirros-kubevirt-vm                       Deleted            Signaled Deletion
2026-10-25T13:00:34Z   cirros-kubevirt-vm                       SuccessfulDelete   Stopped the virtual machine by deleting the virtual machine instance 3f3ff710-5253-4ef9-b573-4aab
2026-10-25T13:00:39Z   cirros-kubevirt-vm                       SuccessfulCreate   Started the virtual machine by creating the new virtual machine instance cirros-kubevirt-vm
2026-10-25T13:00:40Z   multus-kubevirt-vm                       SuccessfulCreate   Started the virtual machine by creating the new virtual machine instance multus-kubevirt-vm
2026-10-25T13:08:30Z   kubevirt-test-vms-remove                 SuccessfulCreate   Created pod: kubevirt-test-vms-remove-sf2tr
2026-10-25T13:08:31Z   kubevirt-test-vms-remove-sf2tr           AddedInterface     Add eth0 [100.72.157.238/32] from k8s-pod-network
2026-10-25T13:08:37Z   kubevirt-test-vms-remove-sf2tr           Created            Container created
crustgather-job-17446644712 ~> kubectl -n rke2-capm3-virt get helmrelease kubevirt -o jsonpath='{range .status.history[*]}{.version} {.chartVersion} {.appVersion} {.lastDeployed}{"\n"}{end}'
2 0.8.0 1.8.3 2026-10-25T13:08:38Z
1 0.7.0 1.7.0 2026-10-25T10:48:23Z
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get vm -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp,GEN:.metadata.generation,OBS:.status.observedGeneration'
NAME                 CREATED                GEN   OBS
cirros-kubevirt-vm   2026-10-25T13:09:47Z   3     1
multus-kubevirt-vm   2026-10-25T13:09:47Z   3     1
crustgather-job-17446644712 ~>

T3 (13:09:55) - the recreated VMs are Running and Ready. Their VMI is at VM generation 1 (Consequence of T2)

crustgather-job-17446644712 ~> kubectl config use-context workload && kubectl -n kubevirt-tests get vm,vmi,pods -o wide
Switched to context "workload".
NAME                 STATUS    READY   AGE
cirros-kubevirt-vm   Running   True    19h
multus-kubevirt-vm   Running   True    19h

NAME                 PHASE     IP               NODENAME                                      READY   LIVE-MIGRATABLE   PAUSED   AGE
cirros-kubevirt-vm   Running   100.72.157.242   wc-2986208786-rke2-capm3-virt-workload-md-0   True    True              19h
multus-kubevirt-vm   Running   100.72.157.243   wc-2986208786-rke2-capm3-virt-workload-md-0   True    True              19h

NAME                                     READY   STATUS    RESTARTS   AGE
virt-launcher-cirros-kubevirt-vm-2tvck   2/2     Running   0          19h
virt-launcher-multus-kubevirt-vm-79sld   2/2     Running   0          19h
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get vm cirros-kubevirt-vm -o jsonpath='{range .status.conditions[*]}{.lastTransitionTime} {.type}={.status} {.reason}: {.message}{"\n"}{end}'; kubectl --context workload -n kubevirt-tests get vmi cirros-kubevirt-vm -o jsonpath='{.metadata.annotations.kubevirt\.io/vm-generation}{"\n"}'
2026-10-25T13:09:55Z Ready=True :
null LiveMigratable=True :
null StorageLiveMigratable=True :
1
crustgather-job-17446644712 ~>

T4 (13:13:22 -> 13:14:23) - the 1.8.3 virt-controller pods are starting and are writing kubevirt.io/pci-topology-version into the template of the existing VMs. Kustomize-controller is re-applying at 13:15:48. The VMs end up at generation 3 with observedGeneration: 1 (Consequence of T2 and T3)

crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt get pods -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp,STARTED:.status.containerStatuses[0].state.running.startedAt,IMAGE:.spec.containers[0].image' | grep -E 'NAME|virt-controller|virt-operator'
NAME                                CREATED                STARTED                IMAGE
virt-controller-67f5779f8f-bvmzm    2026-10-25T13:13:16Z   2026-10-25T13:13:22Z   registry.suse.com/suse/sles/16.0/virt-controller:1.8.3
virt-controller-67f5779f8f-fzl8b    2026-10-25T13:13:42Z   2026-10-25T13:13:47Z   registry.suse.com/suse/sles/16.0/virt-controller:1.8.3
virt-operator-54bc4df697-rwgvb      2026-10-25T13:08:52Z   2026-10-25T13:09:09Z   registry.suse.com/suse/sles/16.0/virt-operator:1.8.3
virt-operator-54bc4df697-w78nx      2026-10-25T13:09:20Z   2026-10-25T13:09:27Z   registry.suse.com/suse/sles/16.0/virt-operator:1.8.3
crustgather-job-17446644712 ~> kubectl --context workload -n kubevirt-tests get vm cirros-kubevirt-vm -o jsonpath='{range .metadata.managedFields[*]}== {.manager} {.operation} {.subresource} {.time}{"\n"}{.fieldsV1}{"\n"}{end}' | cut -c1-700
== kustomize-controller Apply  2026-10-25T13:15:48Z
{"f:metadata":{"f:labels":{"f:kustomize.toolkit.fluxcd.io/name":{},"f:kustomize.toolkit.fluxcd.io/namespace":{},"f:sylva.org/sylva-managed":{}}},"f:spec":{"f:runStrategy":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:kubevirt.io/domain":{},"f:kubevirt.io/size":{}}},"f:spec":{"f:affinity":{"f:nodeAffinity":{"f:requiredDuringSchedulingIgnoredDuringExecution":{}}},"f:domain":{"f:cpu":{"f:cores":{}},"f:devices":{"f:disks":{},"f:interfaces":{}},"f:resources":{"f:requests":{"f:memory":{}}}},"f:networks":{},"f:volumes":{}}}}}
== virt-controller Update  2026-10-25T13:14:23Z
{"f:metadata":{"f:annotations":{".":{},"f:kubevirt.io/latest-observed-api-version":{},"f:kubevirt.io/storage-observed-api-version":{}},"f:finalizers":{".":{},"v:\"kubevirt.io/virtualMachineControllerFinalize\"":{}}},"f:spec":{"f:template":{"f:metadata":{"f:annotations":{".":{},"f:kubevirt.io/pci-topology-version":{}}}}}}
== virt-controller Update status 2026-10-25T13:15:48Z
{"f:status":{".":{},"f:conditions":{},"f:created":{},"f:desiredGeneration":{},"f:observedGeneration":{},"f:printableStatus":{},"f:ready":{},"f:runStrategy":{},"f:volumeSnapshotStatuses":{}}}
crustgather-job-17446644712 ~>

Also the manifest of the unit is identical between 1.8 and main

[git:main]root@tibi-vbmh:sylva-core# git pull
remote: Enumerating objects: 163, done.
remote: Counting objects: 100% (103/103), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 47 (delta 32), reused 35 (delta 22), pack-reused 0 (from 0)
Unpacking objects: 100% (47/47), 9.98 KiB | 150.00 KiB/s, done.
From https://gitlab.com/sylva-projects/sylva-core
   68ef2e1bf..48d9fe9a8  ba/test-branch                      -> origin/ba/test-branch
 * [new branch]          cherry-pick-b1b73d16                -> origin/cherry-pick-b1b73d16
 * [new branch]          dg/kyverno-cel-policies-part2       -> origin/dg/kyverno-cel-policies-part2
   da0dde49e..79079d742  mohan/create-xfs-sc-for-nfs-ganesha -> origin/mohan/create-xfs-sc-for-nfs-ganesha
Already up to date.
[git:main]root@tibi-vbmh:sylva-core# git diff origin/release-1.8 origin/main --stat -- kustomize-units/kubevirt-test-vms/
[git:main]root@tibi-vbmh:sylva-core#

T5 (13:24:18 -> 13:43:50) - kubevirt-test-vms fails its healthcheck with both VMs InProgress and wkld-update-test-units times out after 10m (consequence of T4)

crustgather-job-17446644712 ~> kubectl --context management -n rke2-capm3-virt get kustomization kubevirt-test-vms -o jsonpath='{range .status.conditions[*]}{.lastTransitionTime} {.type}={.status} {.reason}: {.message}{"\n"}{end}'
2026-10-25T13:24:18Z Reconciling=True ProgressingWithRetry: Running health checks for revision 0.0.0-git-4352f62c@sha256:838e76a51ec74872687ebec7ed056febad0f04de7a5f48e9e155685fa5a6601c with a timeout of 30s
2026-10-25T13:24:18Z Ready=False HealthCheckFailed: health check failed after 30.033155722s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026-10-25T13:24:18Z Healthy=False HealthCheckFailed: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
crustgather-job-17446644712 ~>

From the job log of https://gitlab.com/sylva-projects/sylva-core/-/jobs/17446644731

2026/10/25 13:40:12.475075 Kustomization/kubevirt-test-vms state changed: HealthCheckFailed - health check failed after 30.029382528s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026/10/25 13:41:12.553761 Kustomization/kubevirt-test-vms state changed: Progressing - Reconciliation in progress
2026/10/25 13:41:42.988387 Kustomization/kubevirt-test-vms state changed: HealthCheckFailed - health check failed after 30.032704132s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026/10/25 13:42:43.094293 Kustomization/kubevirt-test-vms state changed: Progressing - Reconciliation in progress
2026/10/25 13:43:13.607384 Kustomization/kubevirt-test-vms state changed: HealthCheckFailed - health check failed after 30.03270828s: timeout waiting for: [VirtualMachine/kubevirt-tests/cirros-kubevirt-vm status: 'InProgress', VirtualMachine/kubevirt-tests/multus-kubevirt-vm status: 'InProgress']
2026/10/25 13:43:50.045678 Command timeout exceeded: Kustomization/rke2-capm3-virt/sylva-units-tests-status did not become Ready after 10m0s

The unit uses wait: true with no healthCheckExprs :

[git:main]root@tibi-vbmh:sylva-core# git grep -n -A25 '^  kubevirt-test-vms:' origin/main -- charts/sylva-units/values.yaml | grep -E 'wait|healthCheck|path:'
origin/main:charts/sylva-units/values.yaml-1354-      path: ./kustomize-units/kubevirt-test-vms
origin/main:charts/sylva-units/values.yaml-1355-      wait: true
[git:main]root@tibi-vbmh:sylva-core#

Why kubevirt-test-vms-remove is not enough here:

It actually recreates the VMs while the HelmRelease of KubeVirt is being upgraded (13:08:30 vs 13:08:38), but the new virt-controller only becomes active ~ 5 minutes later (13:13:22) and then is modifying the freshly recreated VMs. ANy restart/recreate done before the new virt-controller is active is undone by it and ordering the remove unit after kubevirt would not reliable either, since the HelmRelease is Ready before virt-operator has finished rolling the components