Support git clone from alternative mirror remote
Runner Product Manager position as of 2022-12-03
My decision is to not implement this functionality. The critical rationale is as follows:
It's another knob that we need to make users aware of that can affect the product's security. Our guiding principle is to keep these (configuration options that can affect GitLab CI/CD's security) at a minimum, as users may be unaware of the implications of enabling specific configuration options.
To add more context to the above guiding principle, we have invested significant development time in strengthening the security posture of GiLab CI/CD and Runner by removing configuration options or changing functionality in the past few releases. Given that experience, we must be incredibly thoughtful about introducing new configuration options.
Overview
I opened this issue to discuss the implementation in !2713 (comment 566279311) to perhaps understanding the problem better and potentially discuss alternative implementations.
Problems:
Gitlab CI, when run at scale, create a lot of read-only workload over the underlying git storage system (Gitaly).
For big pipelines with hundred of jobs being run concurrently in the same stage, that is translated to hundred of concurrent git-ls-remote
and git-fetch
operations with packfiles (blob/tree/commit) being downloaded from a single repository (single Gitlab/Gitaly instance). This limit the scalability of the pipeline by the traffic through put of a single Gitaly instance.
Moreover, if the CI infrastructure(gitlab-runner hosts) is geographically distant compare to the git storage infra(Gitlab / Gitaly), the read-only traffic would also need to be served through high latency network.
Propose solution:
To better solve the requirements in scaling up read-only activities, it would be nice if Gitlab Runner could let end users to define an alternative URL as git remote to fetch objects from during CI.
This would allows end users to run mirror hosts, i.e. using git-daemon or Git CDN, to improve the scalability of CI Pipeline.