Speed up git clone in merge-train
Context
merge-train is used to sync between different GitLab component mirrors (canonical/dev/security), when mirroring is broken in the middle of a patch release process. Thus, its speed it crucial to roll out new change to GitLab.com and to the patch release process.
The step taking the most time is ... git clone, since a full cloning is used. It is explained in a comment:
# We can not perform a shallow clone, as this results in Git sometimes
# refusing to fetch from CE. To work around this, we perform a full clone
# but cache the repository in CI. If a cached repository exists, we simply
# just pull from `master`.Problem Statement
git clone is taking longer and longer, in turn is the whole merge-train run. gitlab-org/release-tools!4494 (merged) mentioned a merge-train run taking 17-21 minutes.
This issue is created to find a solution to speed up git clone in merge-train.
Ideas 💡
Some options:
- Reduce compression to speed up on a fast network connection, e.g.
git config --global core.compression 1 - Make sure to use Git Protocol v2
git config --global protocol.version 2 - If cloning using HTTPS, adapt HTTP buffer size, e.g.
git config --global http.postBuffer 524288000 # 500MB - Increase checkout workers, e.g.
git config --global checkout.workers 8 # Adjust to CPU cores