Skip to content

Expire project caches once per push instead of once per ref

Stan Hu requested to merge sh-post-receive-cache-clear-once into master

Previously ProjectCacheWorker would be scheduled once per ref, which would generate unnecessary I/O and load on Sidekiq, especially if many tags or branches were pushed at once. ProjectCacheWorker would expire three items:

  1. Repository size: This only needs to be updated once per push.
  2. Commit count: This only needs to be updated if the default branch is updated.
  3. Project method caches: This only needs to be updated if the default branch changes, but only if certain files change (e.g. README, CHANGELOG, etc.).

Because the third item requires looking at the actual changes in the commit deltas, we schedule one ProjectCacheWorker to handle the first two cases, and schedule a separate ProjectCacheWorker for the third case if it is needed. As a result, this brings down the number of ProjectCacheWorker jobs from N to 2.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046

Edited by Stan Hu

Merge request reports