Backend: Remove N+1 for Gitaly requests when fetching `includes`
Before being able to increase the number of includes
while properly monitoring performance, we should look into some possible N+1s.
Context:
We have Gitaly N+1 requests because each include
makes one request to the the file's contents.
You can observe in the following dashboard that the ..Config::Process
usually takes an average of 3s
, but there are several spikes where it goes up to 12s
.
@mbobin did some local debugging, and found that the GitLab project uses 31 includes
and the Config::Process
was taking about 5-6 seconds.
Implementation
Work Type | Description | Issue link |
---|---|---|
NOTE: |
||
backend | Backend: The gitlab-ci.yml is limited to 100 includes | #207270 (closed) |
backend | Backend: Remove N+1 for Gitaly requests when fetching includes |
|
backend frontend | Improve the error messaging when fetching remote includes are timing out | #351168 (closed) |
backend | Backend: Improve CI Linter performance through parallelizing HTTP calls | #351250 (closed) |
backend | Backend: Caching includes to improve performance when using remote includes | #351252 |
backend | Backend: Batch request calls to Gitaly when fetching include
|
#382531 (closed) |
backend | Backend: Group files by projects in config_file_project_validate_access | #382751 (closed) |
Edited by Mark Nuzzo