Throttle Geo sync operations enqueued by the log cursor
Geo has two synchronization mechanisms. One goes through various "backfill / catch-up" workers - Geo::RepositorySyncWorker
, Geo::FileDownloadDispatchWorker
, etc, with a strict, user-controllable limit on concurrency.
The second mechanism is activated when the secondary receives one of a range of events from the primary - Geo::RepositoryCreatedEvent
, Geo::RepositoryUpdatedEvent
, etc. In these cases, we update the project registry and also schedule a Geo::ProjectSyncWorker
to run immediately, and asynchronously.
This means that when a flurry of activity happens on the primary, the secondary's sidekiq queue grows significantly, leading to higher-than-user-configured levels of sync concurrency against the primary, if the secondary's sidekiq configuration permits it.
We have this second mechanism so that frequently-used projects jump to the front of the synchronization queue, especially when backfill has only just begun, on a Geo node, but perhaps we need to reduce its aggressiveness.