Refactor the retry interface to be generic

What does this MR do?

Follow up from !4420 (comment 1611107126).

The retry interface was prone to copy-pasting and required Run functions to produce side effects in order to extract values or to modify all the implementors of the Retryable interface in such a matter.

The new interface requires a tiny bit of boilerplate from callers constructing the Retry struct,

Why was this MR needed?

To reduce boilerplate, copy pasting and to encourage the easy of use the retry package.

What's the best way to test this MR?

Everything should function as expected. And tests should pass.

What are the relevant issue numbers?

Closes #36957 (closed)