In sylva we're currently following kubernetes release train: In each sylva .y release (where y refers to minor version of semver version `x.y.z`) management cluster supports one (y') version of kubernetes, and workload clusters support 3 versions of k8s (y', y'-1 and y'-2). In management and workload clusters, we're currently imposing the same constraint as kubernetes: we only support upgrade from release y-1, which leads to high operational pressure since all platforms have to be upgraded 3 times per year. On the other hand, only few components of our clusters actually require so frequent upgrades: the kubernetes control planes needs to be upgraded through all kubernetes version, but not the machine deployments (see [k8s skew policy](https://kubernetes.io/releases/version-skew-policy/), same for longhorn that only [supports upgrades from y-1](https://longhorn.io/docs/1.9.1/deploy/upgrade/#upgrade-path-enforcement-and-downgrade-prevention). But appart from that, many components could support direct upgrades from y-2 or even older versions. This epic describes a solution that could be implemented in sylva to achieve such upgrades. ## Managing multiple sylva-units-releases reconciliations Since sylva-units helmrelease can hardly implements multiple reconciliation of a given unit, the easiest way to implement this feature consists in relying on multiple reconciliations, and rely on some saved state to conditionally set the version of components that need to be upgraded to intermediate versions. We could for example have a global flag called for example `transition_from_n-2` that could be used to conditionally set management-cluster k8s_version in values.yaml: ``` k8s_version_short: '{{ .Values.transition_from_n-2 | ternary "1.32" "1.33" }}' ``` In order to determine that state we could compute a sylva-release version that would be saved in sylva-units-status once all units will have reconciled: ``` sylva_release: "1.6" transition_from_n-2: '{{ lookup "v1" "ConfigMap" .Release.Namespace "sylva-units-status" | dig "data" "sylva_release" "1.4" | semverCompare "<1.5" }}' ``` And additionally, we would another extra flag "additional_reconciliation_required" that would instruct sylva-units-operator (or a kyverno policy) to trigger a new upgrade of the chart. ``` additional_reconciliation_required: '{{ .Values.transition_from_n-2 }}' ``` ## Sample workflow to upgrade from 1.4 to 1.6 When sylva-units HelmRelease will reconcile the first time while upgrading from 1.4, `transition_from_n-2` will evaluate to `true`, so `k8s_version_short` will be set to `1.32` And once all units will have reconciled, sylva-units-status unit will set in the sylva-units-status configMap: ``` sylva_release: 1.6 additional_reconciliation_required: true ``` following this flag, a new reconciliation of sylva-units helmrelease will be initiated, and that time the computed values will be the final ones for sylva 1.6: ``` transition_from_n-2: false sylva_release: 1.6 k8s_version_short: "1.33" additional_reconciliation_required: false ``` Note that the approach is simplified for the purpose of the explanation, we will probably introduce some helpers and intermediate values to provide a more refined/polished interface. ## Computing the current sylva release version As we've seen in the above example, we'll need to be able to determine the current version to compare it with the installed one. Determining this version is fairly easy in production deployments using the tag of current GitRepository/OCIRepository used for sylva-core, but we don't have the counterpart in CI and development environments deployed from various commits of our stack. To overcome that issue, we'll probably have to store the next sylva version in main branch (`sylva_release: 1.6-pre` for example), and the actual version in stable branches (`sylva_release: 1.5` in branch release-1.5). This way we would be able to test the above scenario to migrate from release-1.4 to main in CI. ## Handling the depreciation and pruning of transitional units In past sylva releases, we've often been relying on transitional units (some of them are implemented as one_shot units that only need to run once) One example is the recent migration from `keycloak-postgres` unit to `keycloak-postgresql` during the transition from sylva 1.4 to sylva 1.5 (in order to reconfigure the underlying cnpg cluster) `keycloak-postgres` unit needs anyway to remain in sylva 1.5 code base, otherwise the unit (and the underlying database) would be pruned as soon as sylva-units reconciles in the newer version (this legacy unit will be disabled during the next reconciliation of sylva-units following its enabled_conditions) If we want to migrate straight from sylva 1.4 to sylva 1.6, this unit will also need to remain in sylva 1.6 codebase for the same reason. It means that we'll only be able to remove this legacy unit from sylva codebase once release-1.6 branch will be forked. As that time it will be hard to recall that this unit can be pruned from our values, that's why it would be relevant to add some metadata to properly track such units, for example we could add: ``` keycloak-postgres: info: transitional-unit: from-1.4 ``` This would help identifying which units can be pruned in a given release.