Use exponential backoff for all retries
Currently retries are performed with fixed interval. If something is not right, exponential backoff works better.
Backoff library code from here https://github.com/kubernetes/apimachinery/blob/master/pkg/util/wait/wait.go can probably be used.
Edited by Mikhail Mazurskiy