sylva-units: remove old "test values" testing, replace by Helm unittest tests

This MR modernizes the way we test sylva-units.

The perspective is that to allow future refactorings of this chart, we need a better test suite (refactoring such as the one needed for sylva-projects#40).

Today we have a mix of:

  • set of "test values" that are under charts/sylva-units/test-values and that are played with helm template with the tools/validation/helm-template-yamllint.sh custom shell script
    • the shell script isn't so great to maintain
    • this is very limited: the only thing check is whether helm template succeeds or fail
      • exception: some test had inline gotpl snippet making actual checks
    • it was relying on symlinks to charts/sylva-units/(management.|workload-cluster.|bootstrap.|)values.yaml files, which isn't ideal:
      • we had to add a wrapper around helm to avoid having warings about "found symbolic link"
      • multifile search returned many irrelevant results
    • this was the first thing we had in Sylva, most of our tests are here (> 30 tests)
  • unit tests written in the Helm unittest framework
    • this is a very clean an efficient framework
    • outputs can be checked (of course...)
    • lookup calls can be emulated
    • when can expect it to be faster soon, work being in progress to support parallel execution of tests (https://github.com/helm-unittest/helm-unittest/pull/906)
    • we are writing the new tests there, but we haven't a lot of them today (12 tests)

🌱 This MR translates all the older tests into Helm unittest tests, so this gives us an homogeneous and richer test suite for sylva-units.

As a side-effect, the tools/validation/helm-template-yamllint.sh has been updated/simplified.

Review

See notes below on how this MR was developed.

💡 Although the stack of commits isn't perfectly clean, this MR will be a bit nicer to review one commit at a time.

Other things

  • this MR removes the YAML linting of the Helm output, which hasn't a lot of value (the helm unittest framework is already doing Yaml parsing on this output, and we don't care about details line indentation/spacing etc.).
  • a small improvement was done in tools/oci/build-sylva-units-artifact.py to avoid requiring git (which is not in the container image used by the CI job that now runs it indirectly via helm-test-sylva-units.sh
  • one occurrence of yes as a boolean had to be changed into true in bootstrap.values.yaml because the helm unittest YAML parsing is strict YAML 1.2 and not the "1.1 with bits of 1.2" that the Kubernetes/Helm ecosystem relies on

Development

This rather repetitive work was done thanks to agentic AI:

  • plan/build workflow

  • task definition was, approximately:

    • for each directory under charts/sylva-units/test-values write a test under charts/sylva-units/tests using the same values
      • in a first commit these values are kept in a parallel ..._values.yaml file
      • in a second commit the agent was asked to inline them in the _test.yaml file, when they weren't too large
  • along the conversation, other instructions were given (approximately:)

    • values file common to multiple tests were kept under sylva-units/tests-values
    • add basic asserts matching the goal relevant for the values changed in a test
    • rewrite some of the Helm if/fail or require snippets written in test values as helm unittest asserts statements
    • the test suite need to pass
    • update helm-template-yamllint.sh:
      • rename it helm-test-sylva-units.sh
      • replace the loop on test-values directory by a call to helm unittest
      • keep the part that generates the dynamic files (use-oci-artifacts-final.values.yaml and all units enabled)
      • merge the sylva-units:helm-template-yamllint and the sylva-units:helm-unittest gitlab CI jobs
    • add test corresponding to all-units-enabled.yaml
    • add tests for each individual (management.|workload-cluster.|bootstrap.|)values.yaml file
  • the agent proposed to remove in one place the helm wrapper that we had introduced to hide the "found symbolic link" warning, I asked it to do it everywhere

  • at many moments, I adjusted/edited a few things, and committed the intermediate steps to git

  • the agent discovered this on its own:

    charts/sylva-units/tests/all-units-enabled_test.yaml: uses the chart's existing built-in ci_enable_all_units: true flag (discovered in _helpers.tpl, already used by sylva-ca-dependency_test.yaml) combined with env_type: ci and default_storage_class: cephfs-csi, to force-enable every unit and assert the render succeeds (notFailedTemplate). This requires no generated/static values file — everything is expressed as a simple set: block, self-maintaining since it relies on the chart's own mechanism rather than deriving/duplicating the unit list.

    • as far as I can say, this is valid -- the possibility of this cleanup had been missed in the past

Review

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon Meaning Available values
☁️ Infra Provider capd, capo, capm3
🚀 Bootstrap Provider kubeadm (alias kadm), rke2, okd, ck8s
🐧 Node OS ubuntu, suse, na, leapmicro
🛠️ Deployment Options Deployment option list and description
🎬 Pipeline Scenarios Available scenario list and description
🟢 Enabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🔴 Disabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🏗️ Target platform Can be used to select specific deployment environment Available platform list and description
⚡ Pipeline control autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way
  • 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

  • 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

  • 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

  • ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

  • ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 leapmicro

  • ☁️ capo 🚀 kadm 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🐧 ubuntu 🟢 neuvector,mgmt:harbor

  • ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.6.x 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ misc 🐧 ubuntu 🟢 mgmt:harbor 🔴 neuvector

  • ☁️ capo 🚀 rke2 🛠️ ha,misc,openbao 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse 🎬 upgrade-from-prev-tag

  • ☁️ capm3 🚀 rke2 🐧 suse

  • ☁️ capm3 🚀 kadm 🐧 ubuntu

  • ☁️ capm3 🚀 ck8s 🐧 ubuntu

  • ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 upgrade-from-prev-release-branch 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🎬 sylva-upgrade 🛠️ ha,misc 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 ck8s 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-from-release-1.5

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-to-main

Global config for deployment pipelines

  • autorun pipelines
  • allow failure on pipelines
  • record sylvactl events

Notes:

  • Enabling autorun will make deployment pipelines to be run automatically without human interaction
  • Disabling allow failure will make deployment pipelines mandatory for pipeline success.
  • if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited by Thomas Morin

Merge request reports

Loading
Loading