[backport-1.7]: too many kube-jobs pods are retained

A solution for issue #4569 (closed) needs to be backported to 1.7.x