Rename and revisit _internal.sylva_namespaces computation and tests

What does this MR do and why?

What

What I'm proposing as actual changes to current behavior, on top of the _internal value rename is:

  • keep static list part of ._internal.sylva_namespaces_not_labeled_by_flux for only operator-owned namespaces and avoid including K8s initial namespaces like default & kube-system
  • have the python test exempt known (k8s initial, rancher/fleet ad-hoc, w-c-o) namespaces and thus allow the test to check that all others have the sylva.org/sylva-managed: "true" label.

If you think the default & kube-system need to be kept in ._internal.sylva_namespaces_not_labeled_by_flux I can revert this.

Why

Learnt during this MR's initial state review that ._internal.sylva_namespaces is a small complement to all namespaces (manifests) deployed by Flux (getting sylva.org/sylva-managed: "true" labels directly via commonMetadata.labels), covering only HelmRelease.spec.targetNamespace from units with HelmRelease.spec.install.createNamespace: true and the static list of operator-created (CNI and Rancher only AFAICS) namespaces, not a derivation of "every ns sylva-units deploys".

What it does

  1. Renames the ._internal.sylva_namespaces to ._internal.sylva_namespaces_not_labeled_by_flux to make that contract explicit and adds a doc comment to mention commonMetadata.labels implicit injection by Flux controllers.

  2. Tries to improve this ._internal.sylva_namespaces_not_labeled_by_flux (currently feeding the namespace allow-list consumed by the sylva-namespaces-label Kyverno policy) so it is more accurate by:

    2.1 conditioning the namespace generated by Calico CNI (vs Cilium which is deployed in kube-system, not an operator-created ns, but part of the https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces), dropping tigera-operator ns from the static list because it seems it's handled by Flux directly:

    capo-mgmt|default] ☸ capo-mgmt
    ➜  k get ns tigera-operator --show-managed-fields -o json \
      | jq -r '.metadata.managedFields[]
               | select(.fieldsV1["f:metadata"]["f:labels"]["f:sylva.org/sylva-managed"])
               | .manager'
    helm-controller
    
    [capo-mgmt|default] ☸ capo-mgmt 
    ➜  k get ns calico-system --show-managed-fields -o json \
      | jq -r '.metadata.managedFields[]
               | select(.fieldsV1["f:metadata"]["f:labels"]["f:sylva.org/sylva-managed"])
               | .manager'
    background-controller

    2.2 removing kube-system from the static list due to the fact it's part of https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces and I believe the static part of ._internal.sylva_namespaces_not_labeled_by_flux should be for operators-owned ns only. Also, until now it was labeled by Flux but only in some situations, mainly clusters where local-path-provisioner was not specifically disabled, due to local-path-provisioner unit having it as HelmRelease.spec.targetNamespace, with HelmRelease.install.createNamespace: true. Since kube-system is already present on the cluster, I took the decision to drop units.local-path-provisioner.helmrelease_spec.install.createNamespace: true

    2.3 similarly to kube-system, the default namespace is also part of https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#initial-namespaces, so I dropped it from the static list following the same reasoning.

    2.4 removing cattle-resources-system as it's created by Flux and labeled via commonMetadata.labels, unit rancher-backup-restore-operator-init

    2.5 removing cattle-dashboards as it's created by Flux and labeled via commonMetadata.labels, from monitoring unit's rancher-monitoring helm template, given default chart values:

    defaultDashboards:
      namespace: cattle-dashboards
      useExistingNamespace: false

    Note 1: the local namespace is Rancher-generator, so it remains in the static list (dedicated to operators-owned ns):

    [capo-mgmt|default] ☸ capo-mgmt
    ➜  kubectl get ns local -o jsonpath='{.metadata.ownerReferences}' | jq .
    [
      {
        "apiVersion": "management.cattle.io/v3",
        "kind": "Cluster",
        "name": "local",
        "uid": "1c8fa3c8-daa2-459a-8c43-b316a12e4187"
      }
    ]

    Note 2: even though it is an operator-owned ns, we're not considering workload-cluster-operator per-workload cluster namespaces (<team>--<cluster>) because we can't predict their names, since team/cluster definitions can arrive via the workload-teams-repo GitOps path, so that would not be available for s-u at render time.

  3. Refactors test_sylva_namespaces_are_labeled test, which was before expected to fail (due to https://docs.pytest.org/en/stable/how-to/skipping.html#xfail-mark-test-functions-as-expected-to-fail use) due to existing Sylva CI namespaces that can't be expected to be labelled

    • K8s initial namespaces: default, kube-system,kube-public,kube-node-lease
    • Rancher and Fleet ns: c-, p-, c--p-, cluster-fleet-*
    • workload-cluster-operator ns for each workload cluster

We now, post refactoring, have exemptions for the above 3 items, and are able to determine a real miss in our ._internal.sylva_namespaces_not_labeled_by_flux logic if any namespace is found missing sylva.org/sylva-managed: "true" label.

Additionally, we introduce another test_expected_namespaces_carry_label test that is checking the 2 labelling paths (Flux's own commonMetadata.labels and the sylva-namespaces-label Kyverno policy) work, by confirming the sylva.org/sylva-managed: "true" label presence when the ns is either:

  • managed by Flux - the unit is Kustomization/HelmRelease (ns would have the another label key like kustomize.toolkit.fluxcd.io/name or helm.toolkit.fluxcd.io/name)
  • listed in _internal.sylva_namespaces_not_labeled_by_flux

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon Meaning Available values
☁️ Infra Provider capd, capo, capm3
🚀 Bootstrap Provider kubeadm (alias kadm), rke2, okd, ck8s
🐧 Node OS ubuntu, suse, na, leapmicro
🛠️ Deployment Options Deployment option list and description
🎬 Pipeline Scenarios Available scenario list and description
🟢 Enabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🔴 Disabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🏗️ Target platform Can be used to select specific deployment environment Available platform list and description
⚡ Pipeline control autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way
  • 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

  • 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

  • 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

  • ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

  • ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 leapmicro

  • ☁️ capo 🚀 kadm 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🐧 ubuntu 🟢 neuvector,mgmt:harbor

  • ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.7.x 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ misc 🐧 ubuntu 🟢 mgmt:harbor 🔴 neuvector

  • ☁️ capo 🚀 rke2 🛠️ ha,misc,openbao🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse 🎬 upgrade-from-prev-tag

  • ☁️ capm3 🚀 rke2 🐧 suse

  • ☁️ capm3 🚀 kadm 🐧 ubuntu

  • ☁️ capm3 🚀 ck8s 🐧 ubuntu

  • ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 upgrade-from-prev-release-branch 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🎬 sylva-upgrade 🛠️ ha,misc 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 ck8s 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-from-release-1.5

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-to-main

Global config for deployment pipelines

  • autorun pipelines
  • allow failure on pipelines
  • record sylvactl events

Notes:

  • Enabling autorun will make deployment pipelines to be run automatically without human interaction
  • Disabling allow failure will make deployment pipelines mandatory for pipeline success.
  • if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited by Bogdan-Adrian Burciu

Merge request reports

Loading
Loading