need race-free update of sylva-units

In a run like https://gitlab.com/sylva-projects/sylva-core/-/jobs/8517048252, we observe the following failure:

trigger reconciliation of sylva-units HelmRelease...
  helmrelease.helm.toolkit.fluxcd.io/sylva-units annotated
2024/12/02 10:57:58.263895 GitRepository/sylva-units                          Updated - ObservedGeneration is 1 whereas Generation is 2
2024/12/02 10:58:10.349848 GitRepository/sylva-units state changed: Progressing - building artifact: new upstream revision 'sha1:7efe12ed990e0cb91dfbcbabfa67abc0b6a66e8c'
2024/12/02 10:58:10.349866  ╭╴Waiting for the following resources to progress:
2024/12/02 10:58:10.349870  │  GitRepository/sylva-units               Progressing - building artifact: new upstream revision 'sha1:7efe12ed990e0cb91dfbcbabfa67abc0b6a66e8c'
2024/12/02 10:58:10.349872  ╰╴╴╴┄
2024/12/02 10:58:10.540756 GitRepository/sylva-units state changed: Updated - ObservedGeneration is 1 whereas Generation is 2
2024/12/02 10:58:10.547823 GitRepository/sylva-units state changed: Updated - lastHandledReconcileAt is 2024-12-02T09:38:56 whereas reconcileRequestedAt is 2024-12-02T10:57:58
2024/12/02 10:58:10.844497 HelmChart/sylva-system-sylva-units state changed: ChartPackageSucceeded - packaged 'sylva-units' chart with version '0.0.0-git+7efe12ed990e.2' and merged values files [charts/sylva-units/values.yaml charts/sylva-units/management.values.yaml]
2024/12/02 10:58:17.029331 Failed to store object Kustomization/sylva-system/cluster: object Kustomization/sylva-system/cluster can't depend on Kustomization/sylva-system/kyverno-policies-ready as it would create a cycle: Kustomization/kyverno-policies-ready depends on Kustomization/cluster-machines-ready, Kustomization/cluster-machines-ready depends on Kustomization/cluster
2024/12/02 10:58:18.596428 Kustomization/sylva-system/refresh-metal3machinetemplates has been deleted
2024/12/02 10:58:18.695265 Kustomization/sylva-system/root-dependency-2 has been deleted
2024/12/02 10:58:18.749277 HelmRelease/sylva-system/root-dependency-2 has been deleted
2024/12/02 10:58:18.835459 HelmChart/sylva-system/sylva-system-root-dependency-2 has been deleted
2024/12/02 10:58:18.947087 HelmRelease/sylva-units state changed: Updated - ObservedGeneration is 2 whereas Generation is 3
2024/12/02 10:58:19.185682 HelmRelease/sylva-units state changed: UpgradeSucceeded - Helm upgrade succeeded for release sylva-system/sylva-units.v3 with chart sylva-units@0.0.0-git+fd343b6542c3.2
2024/12/02 10:58:28.258975 GitRepository/sylva-units state changed: Succeeded - stored artifact for revision 'sha1:7efe12ed990e0cb91dfbcbabfa67abc0b6a66e8c'
2024/12/02 10:58:28.258986 All Done: resource HelmRelease/sylva-system/sylva-units is ready!
waiting for root-dependency-3 to become ready...
2024/12/02 10:58:29.032580 GitRepository/devnull                              Succeeded - stored artifact for revision '0.1.0@sha1:8d8494f10c524b13d50917f682595be0b6a912e4'
2024/12/02 10:58:29.032641 GitRepository/sylva-core                           Updated - ObservedGeneration is 1 whereas Generation is 2
2024/12/02 10:58:29.032644 HelmChart/sylva-system-root-dependency-3           ChartPackageSucceeded - packaged 'devnull' chart with version '0.1.0+8d8494f10c52'
2024/12/02 10:58:29.032648 HelmRelease/root-dependency-3                      InstallSucceeded - Helm install succeeded for release sylva-system/root-dependency.v1 with chart devnull@0.1.0+8d8494f10c52
2024/12/02 10:58:33.815198 GitRepository/sylva-core state changed: Progressing - building artifact: new upstream revision 'sha1:7efe12ed990e0cb91dfbcbabfa67abc0b6a66e8c'
2024/12/02 10:58:33.815223  ╭╴Waiting for the following resources to progress:
2024/12/02 10:58:33.815227  │  GitRepository/sylva-core               Progressing - building artifact: new upstream revision 'sha1:7efe12ed990e0cb91dfbcbabfa67abc0b6a66e8c'
2024/12/02 10:58:33.815230  ╰╴╴╴┄
2024/12/02 10:58:34.052299 GitRepository/sylva-core state changed: Updated - ObservedGeneration is 1 whereas Generation is 2
2024/12/02 10:58:34.060540 GitRepository/sylva-core state changed: Updated - lastHandledReconcileAt is 2024-12-02T10:58:16 whereas reconcileRequestedAt is 2024-12-02T10:58:29
2024/12/02 10:58:44.183766 Kustomization/root-dependency-3 state changed: Updated - ObservedGeneration is -1 whereas Generation is 2
2024/12/02 10:58:44.192253 Kustomization/root-dependency-3 state changed: Updated - ObservedGeneration is 1 whereas Generation is 2
2024/12/02 10:58:44.211918 HelmRelease/root-dependency-3 state changed: Updated - ObservedGeneration is 1 whereas Generation is 2
2024/12/02 10:58:44.211933  ╭╴Waiting for the following resources to progress:
2024/12/02 10:58:44.211937  │  GitRepository/sylva-core                    Updated - lastHandledReconcileAt is 2024-12-02T10:58:16 whereas reconcileRequestedAt is 2024-12-02T10:58:29
2024/12/02 10:58:44.211940  │  HelmRelease/root-dependency-3               Updated - ObservedGeneration is 1 whereas Generation is 2
2024/12/02 10:58:44.211942  ╰╴╴╴┄
2024/12/02 10:58:44.288783 Kustomization/sylva-system/root-dependency-3 has been deleted
2024/12/02 10:58:54.364174 HelmRelease/sylva-system/root-dependency-3 has been deleted
2024/12/02 10:58:54.515253 HelmChart/sylva-system/sylva-system-root-dependency-3 has been deleted
Timed-out waiting for the following resources to be ready:
2024/12/02 11:01:28.365148 Command timeout exceeded

In fact the final version of the sylva-units HelmRelease is 4, and sylvactl wrongly stopped at 3.

What happens is the following:

  • the GitRepository and HelmCharts aren't modified yet
  • (we're on a run where values/secrets aren't modified)
  • we are in this state when sylvactl concludes that 2024/12/02 10:58:28.258986 All Done: resource HelmRelease/sylva-system/sylva-units is ready!
  • the code retrieving the last Helm revision runs and gets 3
  • the watch for root-dependency is hence done on root-dependency-3
  • ... later the HelmRelease is updated, becomes version 4, and the root-dependency disappears

We have discussed this problem with @feleouet, and come to the following ideas:

  • (A) we could ensure that the GitRepository, the HelmChart and the HelmRelease would have a dynamic name (sylva-units-$COMMIT_ID); this idea has the major drawback of requiring more shell to handle the garbage collection to cleanup previous versions...
  • (B) the other possibility is to pursue the work started in !2751 (merged), where the HelmRelease is kept suspended until all its sources are modified and ready :
    • we would suspend the GitRepository+HelmChart+HelmRelease
    • sylvactl would evolve to unsuspend the dependencies when they are reconcilable --resume-reconcilable

We tend to conclude that (B) would be the best option.

Given that the issue observed here happened in a run trying to upgrade from 1.2.1 to main, we're likely to see in the coming days/weeks, until we land a fix, so this is %1.3.0 and priorityhigh

Edited by Thomas Morin