Backend: Caching includes to improve performance when using remote includes
Summary
Some users have been receiving a timeout error when using remote includes
. In an effort to help the CI linter perform better, exploring and implementing the ability for includes
to be cached when using remote includes.
Proposal
Need to validate through a smaller set of tests whether incorporating caching into includes will help with the performance of when external includes are being referenced in gitlab-ci.yml when running a pipeline or if it would adversely affect the use of Redis.
Implementation
Work Type | Description | Issue link |
---|---|---|
NOTE: |
||
backend | Backend: The gitlab-ci.yml is limited to 100 includes | #207270 (closed) |
backend | Backend: Remove N+1 for Gitaly requests when fetching includes
|
#344829 (closed) |
backend frontend | Improve the error messaging when fetching remote includes are timing out | #351168 (closed) |
backend | Backend: Improve CI Linter performance through parallelizing HTTP calls | #351250 (closed) |
backend | Backend: Caching includes to improve performance when using remote includes |
|
backend | Backend: Batch request calls to Gitaly when fetching include
|
#382531 (closed) |
backend | Backend: Group files by projects in config_file_project_validate_access | #382751 (closed) |
Cache the response from the Gitlab::HTTP
request in Gitlab::Ci::Config::External::File::Remote
using a cache key like ci-remote-include::LOCATION
(so, ci-remote-include::https://gitlab.com/another-ci-file.yml
) and set the cache expiry to 1 hour
Edited by Mark Nuzzo