Use FluxCD as GitOps solution for infrastructure workloads

As FluxCD is now a more mature tool, we should revisit our FluxCD setup and address the short comings identified on the POC #24556 (closed)

During the POC we moved some infrastructure components like cert-manager, external-dns to be managed with Flux but this was only done in Pre and Staging environments, however these components are still managed using gitlab-helmfiles in Production. We should aim to make Flux production read so we can use it to deploy all infrastructure component releases with a GitOps approach.

Acceptance Criteria:

• Upgrade FluxCD to the latest version.
• Revisit and simplify FluxCD k8s-mgmt repo structure.
• Implement mechanism to get Helm Release diffs in CI.
• Add integration testing to FluxCD https://gitlab.com/gitlab-com/gl-infra/k8s-mgmt repos.
• Complete migration from Gitlab Helmfiles to FluxCD for partially migrated services like cert-manager and external-dns.
• Write Production Readiness for FluxCD.
• Bootstrap FluxCD in Production clusters.
• Deploy Foundation owned services using FluxCD instead of helm-files.