Geo: GeoBackoffDelay improvements

Description

For a Geo setup, when Geo::Scheduler::SchedulerWorker#pending_resources.empty? == true, we apply GeoBackoffDelay.set_backoff_time!() which imposes a 5 minute 'backoff' period where no new jobs can be processed.

Proposal

I think it's a good idea having logic to prevent a thundering herd, but I'd like to discuss the following:

  • Apply the backoff if the number of jobs to process if over specificfied capacity rather than if the number of jobs to process is empty?
  • Update log entries to include the fact that backoff has been applied, and how long for - https://gitlab.com/gitlab-org/gitlab-ee/issues/8694
  • Make the backoff value configurable - 5 mins is an eternity on a quite system!
  • Surface when the backoff is in affect on the /admin/geo* UI somehow so it's obvious to the user
Edited Dec 04, 2018 by Ash McKenzie
Assignee Loading
Time tracking Loading