too many kube-jobs pods are retained
Summary
On a platform where cluster-machines-ready job was failing since a couple of days, I observed a very important amount of pods in htat namespace (676).
Since we've increased the default backofflimit significatively (to 10.000) to implement a virtual "retry forever" policy. This may cause issues as a failing job may lead to the accumulation of a huge amount of pods that is not desirable.
As an alternative, I tried to configure the job with:
backoffLimit: 5
podFailurePolicy:
rules:
- action: Ignore
onExitCodes:
operator: NotIn
values:
- 0Which results in pods being recreated forever (status.failed index is not filled), but the pods are still kept forever, backoffLimit is just ignored.
We'll probably need to implement a deletingpolicy or (another kube-job) to garbage collect these pods.