timeout on "Trigger reconciliation of units" / Flux Helm controller OOM
In the past days or so we've been seeing a number of cases where the "Trigger reconciliation of units" step times out.
Example from https://gitlab.com/sylva-projects/sylva-core/-/jobs/9189839664:
🎯 Trigger reconciliation of units 03:01
trigger reconciliation of sylva-units HelmRelease...
helmrelease.helm.toolkit.fluxcd.io/sylva-units annotated
2025/02/20 00:13:03.566687 HelmRepository/sylva-units Static object - Ready
2025/02/20 00:13:03.566740 Resuming HelmChart/sylva-system/sylva-system-sylva-units
2025/02/20 00:13:03.566767 HelmChart/sylva-system-sylva-units Suspended
2025/02/20 00:13:03.579037 HelmChart/sylva-system-sylva-units state changed: Updated - ObservedGeneration is 2 whereas Generation is 3
2025/02/20 00:13:03.579139 ╭╴Waiting for the following resources to progress:
2025/02/20 00:13:03.579144 │ HelmChart/sylva-system-sylva-units Updated - ObservedGeneration is 2 whereas Generation is 3
2025/02/20 00:13:03.579146 ╰╴╴╴┄
2025/02/20 00:13:04.356613 HelmChart/sylva-system-sylva-units state changed: Progressing - building artifact: packaged 'sylva-units' chart with version '1.3.3+3' and merged values files [values.yaml management.values.yaml use-oci-artifacts.values.yaml]
2025/02/20 00:13:04.373850 HelmChart/sylva-system-sylva-units state changed: Updated - ObservedGeneration is 2 whereas Generation is 3
2025/02/20 00:13:04.381928 HelmChart/sylva-system-sylva-units state changed: ChartPackageSucceeded - packaged 'sylva-units' chart with version '1.3.3+3' and merged values files [values.yaml management.values.yaml use-oci-artifacts.values.yaml]
2025/02/20 00:13:04.382030 ╭╴Waiting for the following resources to progress:
2025/02/20 00:13:04.382034 │ HelmRelease/sylva-units Suspended
2025/02/20 00:13:04.382036 ╰╴╴╴┄
2025/02/20 00:13:04.382036 Resuming HelmRelease/sylva-system/sylva-units
2025/02/20 00:13:04.405350 HelmRelease/sylva-units state changed: Updated - ObservedGeneration is 4 whereas Generation is 5
2025/02/20 00:13:04.423016 HelmChart/sylva-system-sylva-units state changed: Updated - ObservedGeneration is 3 whereas Generation is 4
2025/02/20 00:13:04.423158 ╭╴Waiting for the following resources to progress:
2025/02/20 00:13:04.423162 │ HelmChart/sylva-system-sylva-units Updated - ObservedGeneration is 3 whereas Generation is 4
2025/02/20 00:13:04.423164 ╰╴╴╴┄
2025/02/20 00:13:04.468191 HelmRelease/sylva-units state changed: SourceNotReady - HelmChart 'sylva-system/sylva-system-sylva-units' is not ready: latest generation of object has not been reconciled
2025/02/20 00:13:04.481545 HelmRelease/sylva-units state changed: SourceNotReady - HelmChart 'sylva-system/sylva-system-sylva-units' is not ready: latest generation of object has not been reconciled
2025/02/20 00:13:04.901854 HelmChart/sylva-system-sylva-units state changed: Progressing - building artifact: packaged 'sylva-units' chart with version '0.0.0-git-2221e375+4' and merged values files [values.yaml management.values.yaml use-oci-artifacts.values.yaml]
2025/02/20 00:13:04.914901 HelmChart/sylva-system-sylva-units state changed: Updated - ObservedGeneration is 3 whereas Generation is 4
2025/02/20 00:13:04.920276 HelmChart/sylva-system-sylva-units state changed: Updated - lastHandledReconcileAt is 2025-02-20T00:13:03 whereas reconcileRequestedAt is 2025-02-20T00:13:04
2025/02/20 00:13:05.125163 HelmRelease/sylva-units state changed: Progressing - Running 'upgrade' action with timeout of 5m0s
2025/02/20 00:13:05.553922 HelmChart/sylva-system-sylva-units state changed: Progressing - building artifact: packaged 'sylva-units' chart with version '0.0.0-git-2221e375+4' and merged values files [values.yaml management.values.yaml use-oci-artifacts.values.yaml]
2025/02/20 00:13:05.568462 HelmChart/sylva-system-sylva-units state changed: Updated - lastHandledReconcileAt is 2025-02-20T00:13:03 whereas reconcileRequestedAt is 2025-02-20T00:13:04
2025/02/20 00:13:05.576657 HelmChart/sylva-system-sylva-units state changed: ChartPackageSucceeded - packaged 'sylva-units' chart with version '0.0.0-git-2221e375+4' and merged values files [values.yaml management.values.yaml use-oci-artifacts.values.yaml]
2025/02/20 00:13:05.576788 ╭╴Waiting for the following resources to progress:
2025/02/20 00:13:05.576793 │ HelmRelease/sylva-units Progressing - Running 'upgrade' action with timeout of 5m0s
2025/02/20 00:13:05.576795 ╰╴╴╴┄
2025/02/20 00:13:37.259357 Kustomization/sylva-system/root-dependency-2 has been deleted
2025/02/20 00:13:47.257649 HelmChart/sylva-system/sylva-system-root-dependency-2 has been deleted
2025/02/20 00:13:47.312101 HelmRelease/sylva-system/root-dependency-2 has been deleted
2025/02/20 00:16:03.050251 Command timeout exceeded
Timed-out waiting for the following resources to be ready:
* HelmRelease/sylva-units: Progressing - Running 'upgrade' action with timeout of 5m0sMemory usage is near OOM (980.9 MiB/1.0 GiB), shutting down)
Edited by Thomas Morin