Kyverno policy to cleanup nodes re-created during machine deprovisionning
What does this MR do and why?
We've observed that policy that was preventing nodes from being re-created by kubelet after the node has been deleted by capi controller was not working as expected, and "ghost" nodes are sometimes re-created anyway.
This MR introduces a cleanup policy that will delete such nodes, using the ConfigMap containing provider-ids that was already used by webhook.
The provider-ids ConfigMap evolves to track the creationTimestamp associated with various nodes, this helps the cleanup policy to figure out if a node is legitimate: if the creationTimestamp associated with providerID matches the creationTimestamp of the node, the node is valid, otherwise it's a ghost node.
The ConfigMap is renamed in order to make sure that cleanup policy will see a ConfigMap with the new format, because surprisingly I've observed that following condition:
conditions:
all:
- key: "{{ lookup(blacklistedIDs, providerID) || target.metadata.creationTimestamp }}"
operator: NotEquals
value: "{{ target.metadata.creationTimestamp }}"
Would trigger nodes deletion if we have providerID = "" in the ConfigMap
This renaming is not problematic, because now the ConfigMap will be properly filled on policy creation and update following mutateExistingOnPolicyUpdate: true parameter, it was not the case before, because policy was not processed by background controller because we were matching operations in policy definition:
match:
any:
- resources:
kinds:
- /v1/Node
operations: # <<< these statements were preventing mutate policy
- CREATE # <<< from being processed by background controller
- UPDATE # <<< as it does not see any operation...
While discussing with @tmmorin we concluded that node-password secrets should be deleted automatically by RKE2 once node is deleted as rke2 manage that cleanup through wrangler.cattle.io/node finalizer. (And this assumption has been verified in my tests)
Related reference(s)
Relates-to #1442 (closed) Relates-to #1443 (closed)