ensure that unit Kustomization can't be ready if the unit HelmRelease is suspended

Close #3377 (closed)

This MR adds healthChecks/healthCheckExprs on all Kustomizations of Helm-based units so that the HelmRelease will never be seen as ready if it is suspended.

There are two separate commits, because a small commit does a refactoring to change how the HelmRelease name is controlled, which was necessary for the rest to be done cleanly.

Note that this MR depends on a cleanup done in remove useless problematic legacy transitional ... (!6695 - merged)

Example

$ flux suspend hr calico
$ apply.sh
...
...
 ✓ Kustomization/capo-network-settings - Resource is ready
 ⏸ HelmRelease/calico - Suspended
⠎⠁ Kustomization/calico - Progressing - Reconciliation in progress
 ✗ Unit timeout exceeded: unit Kustomization/calico did not become ready after 5m0s
Details on Kustomization/calico and related resources:
E0528 17:37:22.350557  266719 request.go:1196] "Unexpected error when reading response body" err="net/http: request canceled (Client.Timeout or context cancellation while reading body)"
IDENTIFIER                        STATUS     REASON      MESSAGE
Kustomization/sylva-system/calico InProgress             Kustomization generation is 1283, but latest observed generation is 1282
╰┄╴┬┄┄[Conditions]
   ├┄╴Reconciling                 True       Progressing Running health checks for revision ks-not-ready-if-hr-suspended@sha1:876cf9adab6a15e3c3ac87b2d19458448aca81f5 with a timeout of 30s
   ├┄╴Ready                       Unknown    Progressing Reconciliation in progress
   ╰┄╴Healthy                     Unknown    Progressing Running health checks for revision ks-not-ready-if-hr-suspended@sha1:876cf9adab6a15e3c3ac87b2d19458448aca81f5 with a timeout of 30s

then:

$  flux resume hr calico
► resuming helmrelease calico in sylva-system namespace
✔ helmrelease resumed
◎ waiting for HelmRelease reconciliation
✔ HelmRelease calico reconciliation completed
✔ applied revision v3.31.400
$ sylvactl watch Kustomization/sylva-system/sylva-units-status
......
 ✓ Kustomization/capo - Resource is ready
 ✓ Kustomization/capo-network-settings - Resource is ready
 ✓ Kustomization/calico - Resource is ready

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon Meaning Available values
☁️ Infra Provider capd, capo, capm3
🚀 Bootstrap Provider kubeadm (alias kadm), rke2, okd, ck8s
🐧 Node OS ubuntu, suse, na, leapmicro
🛠️ Deployment Options light-deploy, dev-sources, ha, misc, maxsurge-0, logging, no-logging, cilium
🎬 Pipeline Scenarios Available scenario list and description
🟢 Enabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🏗️ Target platform Can be used to select specific deployment environment (i.e real-bmh for capm3 )
  • 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

  • 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

  • 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

  • ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

  • ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 leapmicro

  • ☁️ capo 🚀 kadm 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🐧 ubuntu 🟢 neuvector,mgmt:harbor

  • ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.6.x 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ ha,misc,openbao🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse 🎬 upgrade-from-prev-tag

  • ☁️ capm3 🚀 rke2 🐧 suse

  • ☁️ capm3 🚀 kadm 🐧 ubuntu

  • ☁️ capm3 🚀 ck8s 🐧 ubuntu

  • ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 upgrade-from-prev-release-branch 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 sylva-upgrade 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 ck8s 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-from-release-1.5

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-to-main

Global config for deployment pipelines

  • autorun pipelines
  • allow failure on pipelines
  • record sylvactl events

Notes:

  • Enabling autorun will make deployment pipelines to be run automatically without human interaction
  • Disabling allow failure will make deployment pipelines mandatory for pipeline success.
  • if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited by Thomas Morin

Merge request reports

Loading