in 2-step upgrade, delay MachineDeployments update during first step

Thomas Morinrequested to merge
k8s-version-optimize-2-step-upgrade-MDs-second-attempt into main

This MR is meant to address #3902 (closed) to reduce node rolling updates during Kubernetes 2-steps upgrades.

Since MD nodes can lag behind a few versions, we can do the following for a 2-step (example): when upgrading from 1.33 to 1.35, during the 1.33 to 1.34 upgrade it is sufficient to only upgrade the CP nodes, and leave MD Nodes in 1.33, the MD Nodes will then be upgraded to 1.35 at the same time as the CP nodes, on the second step.

The approach taken here is, when we're at the first step of a 2-step upgrade, we set MachineDeployment.spec.paused to true, so that MD nodes are frozen during this step.

This MR:

  • does a minor variable name change (wrong name used in current version of the code)
  • extends the computation of the currently observed version to distinguish CP nodes and MD nodes
  • adds a computation to determine if the update of MachineDeployment should be delayed
  • adjusts cluster-machines-ready check so that during the step where MachineDeployments are purposefully paused, no check is done on them

💡 best reviewed one commit at a time 💡

Testing

In https://gitlab.com/sylva-projects/sylva-core/-/pipelines/2540263656 we see that this works as expected:

Raw events reflecting node creation times (from events.log)

$ grep Machine.*SuccessfulSetNodeRef events.log |grep sylva-system
2026-05-20T12:00:59Z    2026-05-20T12:00:59Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-7rmk4  1       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-2
2026-05-20T12:00:59Z    2026-05-20T12:00:59Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-dw6wx  1       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-1
2026-05-20T12:00:59Z    2026-05-20T12:00:59Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-td6rv  2       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-0
2026-05-20T12:00:59Z    2026-05-20T12:00:59Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-md0-knq2h-ggcfg      1       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-md-0
2026-05-20T13:03:17Z    2026-05-20T13:03:17Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-82m8d  1       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-2
2026-05-20T13:19:39Z    2026-05-20T13:19:39Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-x9zcz  1       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-1
2026-05-20T13:32:31Z    2026-05-20T13:32:31Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-kx2ww  2       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-0
2026-05-20T14:04:52Z    2026-05-20T14:04:53Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-lmpjx  2       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-2
2026-05-20T14:22:04Z    2026-05-20T14:22:04Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-l6l5k  1       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-1
2026-05-20T14:36:56Z    2026-05-20T14:36:56Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-control-plane-qfpb4  1       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-cp-0
2026-05-20T14:44:31Z    2026-05-20T14:44:31Z    machine-controller-     Machine sylva-system    mgmt-2540263656-kubeadm-capm3-virt-md0-mgd6m-zlp6h      2       SuccessfulSetNodeRef    mgmt-2540263656-kubeadm-capm3-virt-management-md-0

History reference

A first attempt was done in !7804 (closed)

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon Meaning Available values
☁️ Infra Provider capd, capo, capm3
🚀 Bootstrap Provider kubeadm (alias kadm), rke2, okd, ck8s
🐧 Node OS ubuntu, suse, na, leapmicro
🛠️ Deployment Options Deployment option list and description
🎬 Pipeline Scenarios Available scenario list and description
🟢 Enabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🔴 Disabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🏗️ Target platform Can be used to select specific deployment environment Available platform list and description
⚡ Pipeline control autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way

  • 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

  • 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

  • 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

  • ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

  • ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse

  • ☁️ capo 🚀 kadm 🎬 sylva-upgrade 🐧 ubuntu

  • ☁️ capm3 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

  • ☁️ capm3 🚀 kadm 🎬 sylva-upgrade 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 suse

Global config for deployment pipelines

  • autorun pipelines
  • allow failure on pipelines
  • record sylvactl events

Notes:

  • Enabling autorun will make deployment pipelines to be run automatically without human interaction
  • Disabling allow failure will make deployment pipelines mandatory for pipeline success.
  • if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited by Thomas Morin

Merge request reports