too many kube-jobs pods are retained

Summary

On a platform where cluster-machines-ready job was failing since a couple of days, I observed a very important amount of pods in htat namespace (676).

Since we've increased the default backofflimit significatively (to 10.000) to implement a virtual "retry forever" policy. This may cause issues as a failing job may lead to the accumulation of a huge amount of pods that is not desirable.

As an alternative, I tried to configure the job with:

  backoffLimit: 5
  podFailurePolicy:
    rules:
    - action: Ignore
      onExitCodes:
        operator: NotIn
        values:
        - 0

Which results in pods being recreated forever (status.failed index is not filled), but the pods are still kept forever, backoffLimit is just ignored.

We'll probably need to implement a deletingpolicy or (another kube-job) to garbage collect these pods.