Use Kyverno mutate policy to add node annotation from base64 label

What does this MR do and why?

This change implements a workaround for the lack of CABPR ability to inject node annotations, as defined by #417 (comment 1566487272).
It is making use of sylva-capi-cluster branch add-rke2-node-labels in MR sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster!161 (merged) and addresses the findings described in #567 (closed) for the Kyverno tool to permit mutating a K8s Node resource, plus the idea described inside docs/longhorn.md.

• What is not needed for the purpose defined here, but we might want to keep in one form or another (asking how: a CI job?, fiting in *-capo-misc-* one?):
  • change in .gitlab/ci/templates-deployments.yml is only for showing off, to have a way to ensure the thing worked. It did so in jobs/5269353162 and jobs/5269353117. If we make node-annotation-from-label unit enablement conditioned by enablement of longhorn, this test would not allow for learning if the feature works, since longhorn unit is by default disabled.
  • the values

  rke2:
    nodeLabels:
      sylva.org/annotate-node-from-label: "true"
      node.longhorn.io/create-default-disk: "config"
    node_annotations:
      node.longhorn.io/default-disks-config: '[ { "path":"/var/lib/longhorn", "allowScheduling":true}, { "name":"fast-ssd-disk", "path":"/mnt/extra", "allowScheduling":false, "storageReserved":10485760, "tags":[ "ssd", "fast" ] }]'
      node.longhorn.io/default-node-tags: '["fast","storage"]'

for mgmt and sample workload cluster from environment-values/rke2-capo/values.yaml

• CI outputs:

ubuntu@telcocloud-runner:~$ docker exec -it a996b072bb56 sh
/ # cd builds/sylva-projects/sylva-cre
/builds/sylva-projects/sylva-core # kubectl get node
NAME                                           STATUS   ROLES           AGE   VERSION
bootstrap-1033165612-rke2-capo-control-plane   Ready    control-plane   28m   v1.25.3
/builds/sylva-projects/sylva-core # kubectl --kubeconfig management-cluster-kubeconfig get node
NAME                                                          STATUS   ROLES                       AGE   VERSION
management-cluster-1033165612-rke2-capo-cp-55eeae9189-2lcvc   Ready    control-plane,etcd,master   15m   v1.24.12+rke2r1
management-cluster-1033165612-rke2-capo-cp-55eeae9189-k8rfs   Ready    control-plane,etcd,master   22m   v1.24.12+rke2r1
management-cluster-1033165612-rke2-capo-cp-55eeae9189-pq7qj   Ready    control-plane,etcd,master   12m   v1.24.12+rke2r1  # no idea yet why there's no MD node, some other people pipelines have them
/builds/sylva-projects/sylva-core # kubectl --kubeconfig management-cluster-kubeconfig describe node management-cluster-103316
5612-rke2-capo-cp-55eeae9189-k8rfs
Name:               management-cluster-1033165612-rke2-capo-cp-55eeae9189-k8rfs
Roles:              control-plane,etcd,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=management-cluster-1033165612-rke2-capo-cp-55eeae9189-k8rfs
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/etcd=true
                    node-role.kubernetes.io/master=true
                    node.longhorn.io/create-default-disk=config
                    node.longhorn.io/default-disks-config0=WyB7ICJwYXRoIjoiL3Zhci9saWIvbG9uZ2hvcm4iLCAiYWxsb3dTY2hlZHVsaW5
                    node.longhorn.io/default-disks-config1=nIjp0cnVlfSwgeyAibmFtZSI6ImZhc3Qtc3NkLWRpc2siLCAicGF0aCI6Ii9tbn
                    node.longhorn.io/default-disks-config2=QvZXh0cmEiLCAiYWxsb3dTY2hlZHVsaW5nIjpmYWxzZSwgInN0b3JhZ2VSZXNlc
                    node.longhorn.io/default-disks-config3=nZlZCI6MTA0ODU3NjAsICJ0YWdzIjpbICJzc2QiLCAiZmFzdCIgXSB9XQ-x-x
                    node.longhorn.io/default-node-tags=WyJmYXN0Iiwic3RvcmFnZSJd
                    sylva.org/annotate-node-from-label=true
                    sylva.org/annotate-node-from-label-done-by=kyverno
                    topology.cinder.csi.openstack.org/zone=dev-az
Annotations:        cluster.x-k8s.io/cluster-name: management-cluster-1033165612-rke2-capo
                    :
                    node.longhorn.io/default-disks-config:    # <==
                      [ { "path":"/var/lib/longhorn", "allowScheduling":true}, { "name":"fast-ssd-disk", "path":"/mnt/extra", "allowScheduling":false, "storageR...
                    node.longhorn.io/default-node-tags: ["fast","storage"]    # <==
                    policies.kyverno.io/last-applied-patches:    # <==
                      annotate.annotate-node-from-label-list.kyverno.io: added /metadata/annotations/node.longhorn.io~1default-node-tags

Related reference(s)

Closes #567 (closed)

Test coverage

Some CI testing as mentioned above.