fix(operator): handle already deleted K8s nodes in per-node agent cleanup
Already deleted K8s nodes causes punch of 404 over the operator with the following cases:
-
when a k8s node is deleted white its NeutronOVNAGent CR still exists the neutron-operator fails in _compile_full_state(), as it reads the node to check the disruption budgets and it gets a 404 and the whole NeutronDeployment reconcile stops. The operator retries, failes and never gets past the ovn_agents state, so nothing after it is reconciled anymore.
Hence we treat the missing node as a node without labels. No distruption budget can match, so the stale CR gets delteed. Other API errors than 404 are still raised.
-
Deleting the CR alone is not enough as the neutron-ovn-operator executes delete_resource(), which oatches the node for the l2 migration label and reads it for the l2 locks. Both calls are affected by the 404 and CR stays in deletion. Hence we are handling the 404 there too, so without a node is no label to set and no agent that can hold an l2 lock.
Logging those cases were logged as debug level as in big environments this happens often and could cause a noise in logging.
Signed-off-by: Mohamad Mouselli mohamad.mouselli@digits.schwarz
Co-authored-by: Tobias Fischer tobias.fischer@cloudandheat.com