Add Cilium cluster application
Cilium is a container networking provider that can be used to enable support for NetworkPolicy in a cluster.
Supports installation to both GKE (default) and EKS via clusterType variable.
Unfortunately cilium is not working with k3s due to difference in IPAM/CIDR management. I will do more testing around it but for now I think it will be better to disable it on k3s.
I had to introduce 2 new stages to the CI pipeline (cilium_install and cilium_uninstall) to make necessary kube-system pod restarts on GKE. Without those kube-system pods might be unavailable and helm will fail to reach metrics endpoints. Restarts are necessary since cilium's node init daemonset will reconfigure kubelet to use it's CNI plugin and this might cause some pods (notably kube-dns) to use cilium's IPAM.
I'm temporary serving helm charts from the defend's group repository since cilium doesn't have helm repository. Cilium devs aim to deploy helm repository in upcoming major release which supposed to happen before 12.6.
related to gitlab-org/gitlab#14010 (closed)