Concurrency Constraints in Workload Cluster Deletion Due to Default Concurrent Reconcile Limits in Flux Controller

Summary

When attempting to delete multiple workload clusters simultaneously, the operation may be blocked if all available workers in the Flux controller are occupied.

The Helm controller consumes a worker for the duration of any hooks associated with a Helm release. For instance, when a workload cluster is deleted, it triggers the deletion of the associated Helm release "sylva-units," which in turn initiates the removal of several other Helm releases, including those with hooks. This process can require a minimum of two workers for each workload cluster deletion.

Given the default limit of four workers, only two workload clusters can be deleted in parallel. In the case of Kunai, where an additional Helm release is tied to namespace creation, the system can only handle the deletion of a single workload cluster at a time. This limitation can significantly impact operational efficiency during bulk deletions.