apply scripts - avoid race conditions in how sylva-units HelmRelease is updated

Thomas Morinrequested to merge
apply-fixes into main

This MR is needed to address issue #1904 (closed) which is breaking CI for current work in main (upgrades from 1.2.1 to main).

The motivation behind this MR is twofold:

  • a few months ago, I started it with the following in mind:

    On upgrades from 1.1.1, we currently have two places were the sylva-units HelmRelease is patched, which results in multiple reconciliations (with the first one always failing).

    To avoid this, we can modify apply.sh and apply-workload-cluster.sh so that the sylva-units HelmRelease is suspended and then resumed after its sources are updated.

    This will also benefit the general case of sylva-units upgrade, by avoiding the possibility of having a sylva-units HelmRelease reconciliation happening before its source (GitRepository or HelmRepository) has been updated (because we don't control the order in which kustomize build | kubectl apply modifies resources).

  • more recently (this week), we realized that we have a race-condition issue, breaking some CI runs on upgrades from 1.2.1 to main in particular, as described in #1904 (closed), and that a solution would be to suspend resources related to the sylva-units HelmRelease (GitRepository, HelmChart, HelmRelease) and unsuspend them following the dependency chain

This MR implements this solution:

  • when apply.sh (or apply-workload-cluster.sh) is used, the first step is to suspend the sylva-units HelmChart, and HelmRelease
  • once this is done sylvactl is called with the new --resume-suspended option (introduced by sylva-projects/sylva-elements/sylvactl!274 (merged)) that will unsuspend them in the right order, following the dependency chain: unsuspend HelmChart when GitRepository becomes ready, unsuspend HelmRelease when HelmChart becomes ready

This MR depends on sylva-projects/sylva-elements/sylvactl!274 (merged)

This MR does 2 additional things are done in this MR:

  • ensure that --resume-suspended is not set in the context of preview script
  • add a debug-on-exit of the whole manifest of sylva-units HelmRelease (except Secrets)
Edited by Thomas Morin

Merge request reports