Geo: GeoBackoffDelay improvements
Description
For a Geo setup, when Geo::Scheduler::SchedulerWorker#pending_resources.empty? == true
, we apply GeoBackoffDelay.set_backoff_time!()
which imposes a 5 minute 'backoff' period where no new jobs can be processed.
Proposal
I think it's a good idea having logic to prevent a thundering herd, but I'd like to discuss the following:
- Apply the backoff if the number of jobs to process if over specificfied capacity rather than if the number of jobs to process is empty?
- Update log entries to include the fact that backoff has been applied, and how long for - https://gitlab.com/gitlab-org/gitlab-ee/issues/8694
- Make the backoff value configurable - 5 mins is an eternity on a quite system!
- Surface when the backoff is in affect on the /admin/geo* UI somehow so it's obvious to the user
Edited by Ash McKenzie