Throttle Geo sync operations enqueued by the log cursor

Geo has two synchronization mechanisms. One goes through various "backfill / catch-up" workers - Geo::RepositorySyncWorker, Geo::FileDownloadDispatchWorker, etc, with a strict, user-controllable limit on concurrency.

The second mechanism is activated when the secondary receives one of a range of events from the primary - Geo::RepositoryCreatedEvent, Geo::RepositoryUpdatedEvent, etc. In these cases, we update the project registry and also schedule a Geo::ProjectSyncWorker to run immediately, and asynchronously.

This means that when a flurry of activity happens on the primary, the secondary's sidekiq queue grows significantly, leading to higher-than-user-configured levels of sync concurrency against the primary, if the secondary's sidekiq configuration permits it.

We have this second mechanism so that frequently-used projects jump to the front of the synchronization queue, especially when backfill has only just begun, on a Geo node, but perhaps we need to reduce its aggressiveness.

/cc @stanhu @jramsay @jarv

Assignee Loading
Time tracking Loading