Improve scheduling of mirror updates to reduce frequency of database queries
What does this MR do?
This aims to reduce the frequency of a database query which takes 50% of total query time on GitLab.com (!), see https://gitlab.com/gitlab-org/gitlab-ce/issues/60524.
There is one actual change and one behind a feature flag:
- Swapping conditions (see below)
- Move the rescheduling logic from the individual
RepositoryUpdateMirrorWorker
toUpdateAllMirrorsWorker
Conditions swap
Swapping the conditions means that the we'll always attempt to obtain the lease (as a side-effect of try_obtain
) even if the reschedule_immediately?
is false
. However, that would probably greatly reduce the frequency of the database query behind reschedule_immediately?
. The drawback is that any job would be delayed for 2s (while waiting for the lease, even in case reschedule_immediately?
is false
).
UpdateAllMirrorsWorker
Rescheduling in Instead of having every RepositoryUpdateMirrorWorker
(which there are many on GitLab.com) job reschedule the UpdateAllMirrorsWorker
, we remove the scheduling logic from the RepositoryUpdateMirrorWorker
job and let the UpdateAllMirrorsWorker
reschedule itself if it detects left-over capacity after waiting for 10s. The wait is to give running jobs a chance to complete before we re-check for capacity.
This is behind a feature flag update_all_mirrors_worker_rescheduling
. The follow-up issue to check and remove the flag is https://gitlab.com/gitlab-org/gitlab-ee/issues/11499.