sylva-units: remove old "test values" testing, replace by Helm unittest tests
This MR modernizes the way we test sylva-units.
The perspective is that to allow future refactorings of this chart, we need a better test suite (refactoring such as the one needed for sylva-projects#40).
Today we have a mix of:
- set of "test values" that are under
charts/sylva-units/test-valuesand that are played withhelm templatewith thetools/validation/helm-template-yamllint.shcustom shell script- the shell script isn't so great to maintain
- this is very limited: the only thing check is whether
helm templatesucceeds or fail- exception: some test had inline gotpl snippet making actual checks
- it was relying on symlinks to
charts/sylva-units/(management.|workload-cluster.|bootstrap.|)values.yamlfiles, which isn't ideal:- we had to add a wrapper around helm to avoid having warings about "found symbolic link"
- multifile search returned many irrelevant results
- this was the first thing we had in Sylva, most of our tests are here (> 30 tests)
- unit tests written in the Helm unittest framework
- this is a very clean an efficient framework
- outputs can be checked (of course...)
lookupcalls can be emulated- when can expect it to be faster soon, work being in progress to support parallel execution of tests (https://github.com/helm-unittest/helm-unittest/pull/906)
- we are writing the new tests there, but we haven't a lot of them today (12 tests)
unittest tests, so this gives us an homogeneous and richer test suite for sylva-units.
As a side-effect, the tools/validation/helm-template-yamllint.sh has been updated/simplified.
Review
See notes below on how this MR was developed.
Other things
- this MR removes the YAML linting of the Helm output, which hasn't a lot of value (the helm unittest framework is already doing Yaml parsing on this output, and we don't care about details line indentation/spacing etc.).
- a small improvement was done in
tools/oci/build-sylva-units-artifact.pyto avoid requiringgit(which is not in the container image used by the CI job that now runs it indirectly viahelm-test-sylva-units.sh - one occurrence of
yesas a boolean had to be changed intotruein bootstrap.values.yaml because thehelm unittestYAML parsing is strict YAML 1.2 and not the "1.1 with bits of 1.2" that the Kubernetes/Helm ecosystem relies on
Development
This rather repetitive work was done thanks to agentic AI:
-
plan/build workflow
-
task definition was, approximately:
- for each directory under
charts/sylva-units/test-valueswrite a test undercharts/sylva-units/testsusing the same values- in a first commit these values are kept in a parallel
..._values.yamlfile - in a second commit the agent was asked to inline them in the
_test.yamlfile, when they weren't too large
- in a first commit these values are kept in a parallel
- for each directory under
-
along the conversation, other instructions were given (approximately:)
- values file common to multiple tests were kept under
sylva-units/tests-values - add basic asserts matching the goal relevant for the values changed in a test
- rewrite some of the Helm
if/failorrequiresnippets written in test values as helm unittestassertsstatements - the test suite need to pass
- update
helm-template-yamllint.sh:- rename it
helm-test-sylva-units.sh - replace the loop on test-values directory by a call to helm unittest
- keep the part that generates the dynamic files (use-oci-artifacts-final.values.yaml and all units enabled)
- merge the sylva-units:helm-template-yamllint and the sylva-units:helm-unittest gitlab CI jobs
- rename it
- add test corresponding to all-units-enabled.yaml
- add tests for each individual
(management.|workload-cluster.|bootstrap.|)values.yamlfile
- values file common to multiple tests were kept under
-
the agent proposed to remove in one place the helm wrapper that we had introduced to hide the "found symbolic link" warning, I asked it to do it everywhere
-
at many moments, I adjusted/edited a few things, and committed the intermediate steps to git
-
the agent discovered this on its own:
charts/sylva-units/tests/all-units-enabled_test.yaml: uses the chart's existing built-in
ci_enable_all_units: trueflag (discovered in _helpers.tpl, already used by sylva-ca-dependency_test.yaml) combined withenv_type: cianddefault_storage_class: cephfs-csi, to force-enable every unit and assert the render succeeds (notFailedTemplate). This requires no generated/static values file — everything is expressed as a simple set: block, self-maintaining since it relies on the chart's own mechanism rather than deriving/duplicating the unit list.- as far as I can say, this is valid -- the possibility of this cleanup had been missed in the past
Review
CI configuration
Below you can choose test deployment variants to run in this MR's CI.
Click to open to CI configuration
Legend:
| Icon | Meaning | Available values |
|---|---|---|
| Infra Provider | capd, capo, capm3 |
|
| Bootstrap Provider | kubeadm (alias kadm), rke2, okd, ck8s |
|
| Node OS | ubuntu, suse, na, leapmicro |
|
| Deployment Options | Deployment option list and description | |
| Pipeline Scenarios | Available scenario list and description | |
| Enabled units | Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type |
|
| Disabled units | Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type |
|
| Target platform | Can be used to select specific deployment environment Available platform list and description | |
| Pipeline control | autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way |
-
🎬 preview☁️ capd🚀 kadm🐧 ubuntu -
🎬 preview☁️ capo🚀 rke2🐧 suse -
🎬 preview☁️ capm3🚀 rke2🐧 ubuntu -
☁️ capd🚀 kadm🛠️ light-deploy🐧 ubuntu -
☁️ capd🚀 rke2🛠️ light-deploy🐧 suse -
☁️ capo🚀 rke2🐧 suse -
☁️ capo🚀 rke2🐧 leapmicro -
☁️ capo🚀 kadm🐧 ubuntu -
☁️ capo🚀 kadm🐧 ubuntu🟢 neuvector,mgmt:harbor -
☁️ capo🚀 rke2🎬 rolling-update🛠️ ha🐧 ubuntu -
☁️ capo🚀 kadm🎬 wkld-k8s-upgrade🐧 ubuntu -
☁️ capo🚀 rke2🎬 rolling-update-no-wkld🛠️ ha🐧 suse -
☁️ capo🚀 rke2🎬 sylva-upgrade🛠️ ha🐧 ubuntu -
☁️ capo🚀 rke2🎬 sylva-upgrade-from-1.6.x🛠️ ha,misc🐧 ubuntu -
☁️ capo🚀 rke2🛠️ ha,misc🐧 ubuntu -
☁️ capo🚀 rke2🛠️ misc🐧 ubuntu🟢 mgmt:harbor🔴 neuvector -
☁️ capo🚀 rke2🛠️ ha,misc,openbao🐧 suse -
☁️ capo🚀 rke2🐧 suse🎬 upgrade-from-prev-tag -
☁️ capm3🚀 rke2🐧 suse -
☁️ capm3🚀 kadm🐧 ubuntu -
☁️ capm3🚀 ck8s🐧 ubuntu -
☁️ capm3🚀 kadm🎬 rolling-update-no-wkld🛠️ ha,misc🐧 ubuntu -
☁️ capm3🚀 rke2🎬 wkld-k8s-upgrade🛠️ ha🐧 suse -
☁️ capm3🚀 kadm🎬 rolling-update🛠️ ha🐧 ubuntu -
☁️ capm3🚀 rke2🎬 upgrade-from-prev-release-branch🛠️ ha🐧 suse -
☁️ capm3🚀 rke2🛠️ misc,ha🐧 suse -
☁️ capm3🚀 rke2🎬 sylva-upgrade🛠️ ha,misc🐧 suse -
☁️ capm3🚀 kadm🎬 rolling-update🛠️ ha🐧 suse -
☁️ capm3🚀 ck8s🎬 rolling-update🛠️ ha🐧 ubuntu -
☁️ capm3🚀 rke2|okd🎬 no-update🐧 ubuntu|na -
☁️ capm3🚀 rke2🐧 suse🎬 upgrade-from-release-1.5 -
☁️ capm3🚀 rke2🐧 suse🎬 upgrade-to-main
Global config for deployment pipelines
- autorun pipelines
- allow failure on pipelines
- record sylvactl events
Notes:
- Enabling
autorunwill make deployment pipelines to be run automatically without human interaction - Disabling
allow failurewill make deployment pipelines mandatory for pipeline success. - if both
autorunandallow failureare disabled, deployment pipelines will need manual triggering but will be blocking the pipeline
Be aware: after configuration change, pipeline is not triggered automatically.
Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.