Draft: Switch cluster CNI to Cilium

What does this MR do and why?

Introduces Cilium as default cluster CNI.

For fresh Sylva installation, Cilium will be the default CNI instead of Calico, configured with full functionality: eBPF based routing, kube-proxy-replacement, egress gateway etc.

For clusters already deployed, there will be some manual actions required because the full migration requires 2 steps:

1. On first run of apply.sh:

• Cilium will be deployed along Calico with the required helm values to function in a "latent" mode
• on node rolling upgrade each new node will receive Cilium settings from a CiliumNodeConfig in order to activate Cilium as default CNI, but using legacy routing instead of eBPF in order to work with the remaining Calico nodes.
• after all nodes finish rolling and all units are reconciled the cluster will run on Cilium, but with limited functionality.

2. On second run of apply.sh (only if the previous one completed successfully):

• Calico unit will be disabled (removed)
• (optional until Sylva 1.6) users have to specifically enable kubeProxyReplacement or/and egressGateway which will switch Cilium to run in eBPF mode triggering a new node rolling update, removing kube-proxy from the nodes. For this step, MD nodes need to be updated after CP nodes.

Depends on sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster!588 (closed)

Closes: #2092 (closed) #2346

cc @vladonutu

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	capd, capo, capm3
🚀	Bootstrap Provider	kubeadm (alias kadm), rke2
🐧	Node OS	ubuntu, suse
🛠️	Deployment Options	light-deploy, dev-sources, ha, misc, maxsurge-0, logging, no-logging
🎬	Pipeline Scenarios	Available scenario list and description

• 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

• 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

• 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

• ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

• ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

• ☁️ capo 🚀 rke2 🐧 suse

• ☁️ capo 🚀 kadm 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

• ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse

• ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha,misc 🐧 ubuntu

• ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🐧 suse

• ☁️ capm3 🚀 kadm 🐧 ubuntu

• ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

• ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha 🐧 suse

• ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse

• ☁️ capm3 🚀 rke2 🎬 sylva-upgrade-from-1.4.x 🛠️ ha,misc 🐧 suse

• ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

• ☁️ capm3 🚀 ck8s 🎬 no-wkld 🛠️ light-deploy 🐧 ubuntu

Global config for deployment pipelines

• autorun pipelines
• allow failure on pipelines
• record sylvactl events

Notes:

• Enabling autorun will make deployment pipelines to be run automatically without human interaction
• Disabling allow failure will make deployment pipelines mandatory for pipeline success.
• if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.