Add a mechanism to retry rolling out the namespaces having last_rollout_failed_at
Background
In this issue, we have introduced a metadata last_rollout_failed_at on zoekt_enabled_namespaces. As a simple implementation, the RolloutWorker was just skipping the namespaces which has this metadata set. This is fine for a first iteration, but it will block these namespaces from getting rolling out. We can not keep these namespaces in the blocked state, and we should consider re-trying rolling out these namespaces.
Proposal
Add a mechanism to retry these namespaces with some delay. This delay should be configurable from the admin settings.
Note: Add the setting attribute here as well: https://gitlab.com/gitlab-org/gitlab/-/blob/5a1a34b19ac483a82f430ee493acf99b194405fa/ee/app/models/search/zoekt/settings.rb
Edited by Ravi Kumar