Make sure we don't depend on queue size for workers running on the catchall shard
Sometimes the application wants to know how many jobs are in a queue to limit the amount that is being done concurrently. This could be done on the queue size because a worker would not share it's queue with others. Besides, when we switch from queue-per-worker model, some operational tools or scripts are affected because they use queue name and worker name interchangeably.
We've identified these workers:
Workers | Note |
---|---|
Geo::VerificationBatchWorker |
LimitedCapacity::Worker |
Geo::ReverificationBatchWorker |
LimitedCapacity::Worker |
Ci::DeleteObjectsWorker |
LimitedCapacity::Worker |
ContainerExpirationPolicies::CleanupContainerRepositoryWorker |
LimitedCapacity::Worker |
ElasticCommitIndexerWorker |
|
ElasticDeleteProjectWorker |
|
UpdateAllMirrorsWorker |
The ProjectImportScheduleWorker needs to keep it's own queue in this iteration |
::HashedStorage::RollbackerWorker |
Keep its queue. Related to Gitlab::HashedStorage::Migrator
|
::HashedStorage::ProjectRollbackWorker |
Keep its queue. Related to Gitlab::HashedStorage::Migrator
|
Gitlab::SidekiqQueue |
Keep it as it is. We would need to make some noise around this when we make &194 (closed) or &447 (closed) available to self-managed installations who could also be using this. |
LimitedCapacity::Worker
Solution for As the remaining capacity doesn't to differentiate running jobs, we could do the registration right after the jobs were scheduled. See #995 (comment 547688265) for more details.
UpdateAllMirrorsWorker
#1027 (moved)
Workaround for the The UpdateAllMirrorsWorker
checks the queue size of ProjectImportScheduleWorker
to wait before rescheduling itself and scheduling more. For now, we need to keep the ProjectImportScheduleWorker
on its own queue.
The architecture of this feature is being discussed in gitlab-org/gitlab#325496 (comment 552567063) in the context of improving some badly performing queries.
The decision there might impact how we remove the dependency of UpdateAllMirrorsWorker
on ProjectImportScheduleWorker
's queue size.
HashedStorage
workers #1028 (closed)
Workaround for the These workers aren't running for us any more. We can keep them on their own queues so things keep working for self-managed installs, our catchall fleet does not need to watch them.
Conclusion
Fix LimitedCapacity::Worker #1035 (closed), leave the rest in dedicated queues for now.