Manage the Longhorn engines upgrade

What does this MR do and why?

TODO

1. set concurrentAutomaticEngineUpgradePerNodeLimit value to 3

2. monitor the completion of all old engines upgrade

3. delete the old engine image once it is no more in use

For that purpose, we use a kyverno policy that determines the last image version, and removes all older engineimages with a refcount=0

In order to sort versions, we sort the 3 digits one by one in JMESpatch as following:

$ echo '["v1.5.5","v1.6.2", "v2.20.0", "v1.0.12", "v1.12.5", "v1.0.11", "v1.6.3"]' | kyverno jp query -c "[] | map(&trim_prefix(@, 'v') | split(@, '.'), @) | sort_by(@, &to_number([2])) |sort_by(@, &to_number([1])) | sort_by(@, &to_number([0])) | map(&join('.', @), @)"
Reading from terminal input.
Enter input object and hit Ctrl+D.
# [] | map(&trim_prefix(@, 'v') | split(@, '.'), @) | sort_by(@, &to_number([2])) |sort_by(@, &to_number([1])) | sort_by(@, &to_number([0])) | map(&join('.', @), @)
["1.0.11","1.0.12","1.5.5","1.6.2","1.6.3","1.12.5","2.20.0"]

Related reference(s)

Test coverage

CI configuration

CI pipelines perform an update for both management and workload clusters, this update will NOT perform a ClusterAPI rolling update (deletion and creation of new K8s nodes) by default.

For some cases, it may be relevant to perform more complex tests.

Theses features can be activated in an MR by adding one of these labels to the MR and will apply to the next pipelines.

• adding the label ci-featuretest-rolling-update pipelines will perform a node rolling update in the -update jobs (without version upgrades)
• adding the label ci-featuretest-upgrade-from-1.1.1 pipelines will perform an upgrade from Sylva 1.1.1 to your dev branch (including a k8s version upgrade resulting in a node rolling update)

Closes #1713 (closed)