CI: test-kubeadm-capd - Vault FluxCD HelmRelease stuck on "another operation (install/upgrade/rollback) is in progress"

I've seen lots of cases of test-kubeadm-capd jobs failing, stuck on vault HelmRelease being blocked on "another operation (install/upgrade/rollback) is in progress".

This matches this FluxCD upstream issue: https://github.com/fluxcd/helm-controller/issues/149

The root cause would be an OOM (which is very plausible in our resource-constrained kubead-capd environment).

And recent FluxCD release 0.41 seem to have something to help ( https://github.com/fluxcd/flux2/releases/tag/v0.41.0 )

  • Opt-in OOM watcher in helm-controller to handle graceful termination of the controller before it is forcefully killed by Kubernetes' OOM killer.
  • Kubernetes client and Custom Resource Definition life-cycle improvements to reduce the memory consumption of the helm-controller, with observed reductions up to 50%. ...
  • Optional disabling of the cache of the status poller used to determine the health of the resources applied by the kustomize-controller. This may improve memory usage on large scale clusters at the cost of more direct API calls.