Skip to content

Improve scheduling of mirror updates to reduce frequency of database queries

Andreas Brandl requested to merge ab-60524-high-frequency-query into master

What does this MR do?

This aims to reduce the frequency of a database query which takes 50% of total query time on GitLab.com (!), see https://gitlab.com/gitlab-org/gitlab-ce/issues/60524.

There is one actual change and one behind a feature flag:

  1. Swapping conditions (see below)
  2. Move the rescheduling logic from the individual RepositoryUpdateMirrorWorker to UpdateAllMirrorsWorker

Conditions swap

Swapping the conditions means that the we'll always attempt to obtain the lease (as a side-effect of try_obtain) even if the reschedule_immediately? is false. However, that would probably greatly reduce the frequency of the database query behind reschedule_immediately?. The drawback is that any job would be delayed for 2s (while waiting for the lease, even in case reschedule_immediately? is false).

Rescheduling in UpdateAllMirrorsWorker

Instead of having every RepositoryUpdateMirrorWorker (which there are many on GitLab.com) job reschedule the UpdateAllMirrorsWorker, we remove the scheduling logic from the RepositoryUpdateMirrorWorker job and let the UpdateAllMirrorsWorker reschedule itself if it detects left-over capacity after waiting for 10s. The wait is to give running jobs a chance to complete before we re-check for capacity.

This is behind a feature flag update_all_mirrors_worker_rescheduling. The follow-up issue to check and remove the flag is https://gitlab.com/gitlab-org/gitlab-ee/issues/11499.

Edited by Andreas Brandl

Merge request reports