Skip to content

Backend: Remove N+1 for Gitaly requests when fetching `includes`

Before being able to increase the number of includes while properly monitoring performance, we should look into some possible N+1s.

Context:
We have Gitaly N+1 requests because each include makes one request to the the file's contents.

You can observe in the following dashboard that the ..Config::Process usually takes an average of 3s, but there are several spikes where it goes up to 12s.

https://dashboards.gitlab.net/d/stage-groups-pipeline_execution/stage-groups-group-dashboard-verify-pipeline-execution?viewPanel=1173128221&orgId=1&from=now-7d&to=now

@mbobin did some local debugging, and found that the GitLab project uses 31 includes and the Config::Process was taking about 5-6 seconds.

Implementation

Work Type Description Issue link
NOTE: 🚨 All below issues can be done in parallel
backend Backend: The gitlab-ci.yml is limited to 100 includes #207270 (closed)
backend Backend: Remove N+1 for Gitaly requests when fetching includes 👈 You are here
backend frontend Improve the error messaging when fetching remote includes are timing out #351168 (closed)
backend Backend: Improve CI Linter performance through parallelizing HTTP calls #351250 (closed)
backend Backend: Caching includes to improve performance when using remote includes #351252
backend Backend: Batch request calls to Gitaly when fetching include #382531 (closed)
backend Backend: Group files by projects in config_file_project_validate_access #382751 (closed)
Edited by Mark Nuzzo