ci: Build workhorse binaries and upload/download as generic package
What does this MR do and why?
This MR does two things:
- Adds a new
build-components
job that runs automatically for all GitLab.comgitlab-org/gitlab
scheduled pipelines, for anymaster
commit that touches theworkhorse/
folder, and is manual for GitLab.com'sgitlab-org
's MRs.- This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab's test suite (under
tmp/tests/gitlab-workhorse
). - If the package URL returns a 404:
- It runs
scripts/setup-test-env
, so that the GitLab Workhorse binaries are built. - It then creates an archive which contains the binaries and upload it as a generic package to https://gitlab.com/gitlab-org/gitlab/-/packages/.
- It runs
- This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab's test suite (under
- Change the
setup-test-env
job to:- First download the GitLab Workhorse generic package build and uploaded by
build-components
. - If the package is retrieved successfully, its content is placed in the right folder (i.e.
tmp/tests/gitlab-workhorse
), preventing the building of the binaries whenscripts/setup-test-env
is run later on. - If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of
scripts/setup-test-env
- First download the GitLab Workhorse generic package build and uploaded by
Notes:
- The diff to achieve this is quite small, and we can see that
TestEnv
didn't have to be touched at all, so this is really an optimization that sits on top of what we have today (i.e. caching of GitLab Workhorse binaries). - The version of the package is the
workhorse
tree SHA (i.e.git rev-parse HEAD:workhorse
). - Since the
workhorse
shouldn't change too often, most of the MRs should be able to take advantage of the pre-built package, which would be built every two hours. - As a further iteration, the package could be downloaded by
TestEnv
directly, so that local GDK instances, and GitPod instances (not sure about that) would benefit from it as well (though, different architectures should be taken in account, see gitlab-development-kit#1391).
Improvements
scripts/setup-test-env
Duration of With package | Duration |
---|---|
No | 92s, 84s, 82s |
Yes | 42s, 44s, 46s |
It looks like we could already save 42 seconds on average for the setup-test-env
job.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR. -
Documentation is up-to-date: #353078 (closed) => !81289 (merged)
Related to #351035 (closed).
Edited by Rémy Coutable