Skip to content

Reduce Gitaly calls in PostReceive

Stan Hu requested to merge sh-optimize-commit-deltas-post-receive into master

This commit reduces I/O load and memory utilization during PostReceive for the common case when no project hooks or services are set up.

We saw a Gitaly N+1 issue in CommitDelta when many tags or branches are pushed. We can reduce this overhead in the common case because we observe that most new projects do not have any Web hooks or services, especially when they are first created. Previously, BaseHooksService unconditionally iterated through the last 20 commits of each ref to build the push_data structure. The push_data structured was used in numerous places:

  1. Building the push payload in EventCreateService
  2. Creating a CI pipeline
  3. Executing project Web or system hooks
  4. Executing project services
  5. As the return value of BaseHooksService#execute
  6. Finding files to invalidate the cache in BranchHooksService#invalidated_file_types

We only need to generate the full push_data for items 3, 4, and 6.

Item 1: EventCreateService only needs the last commit and doesn't actually need the commit deltas.

Item 2: In addition, Ci::CreatePipelineService only needed a subset of the parameters.

Item 5: The return value of BaseHooksService#execute also wasn't being used anywhere.

Item 6: This is only used when pushing to the default branch, so if many tags are pushed we can save significant I/O here.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65878

Edited by Stan Hu

Merge request reports