Make UpdateRepositoryStorageWorker retryable

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem

UpdateRepositoryStorageWorker cannot pick up work unless it's in scheduled state. When the worker stops processing because of the interruption (or exception), it restarts but it's unable to continue from the previous point.

That leads to following issues:

• Sidekiq retry functionality makes no sense, because the job is not retryable
• state_machines functionality is not fully used.
• The worker marked as idempotent, but it's not fully true.

Proposal

Refactor code to allow the worker to proceed from any state and be able to complete the work after the retry.

Previous discussions

• https://gitlab.com/gitlab-org/gitlab/-/issues/429049#note_1642657650

• https://gitlab.com/gitlab-org/gitlab/-/issues/429049+

• https://gitlab.com/gitlab-org/gitlab/-/issues/427254+