ci: Build workhorse binaries and upload/download as generic package
What does this MR do and why?
This MR does two things:
- Adds a new
build-components
job that runs automatically for all GitLab.comgitlab-org/gitlab
scheduled pipelines, for anymaster
commit that touches theworkhorse/
folder, and is manual for GitLab.com'sgitlab-org
's MRs.- This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab's test suite (under
tmp/tests/gitlab-workhorse
). - If the package URL returns a 404:
- It runs
scripts/setup-test-env
, so that the GitLab Workhorse binaries are built. - It then creates an archive which contains the binaries and upload it as a generic package to https://gitlab.com/gitlab-org/gitlab/-/packages/.
- It runs
- This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab's test suite (under
- Change the
setup-test-env
job to:- First download the GitLab Workhorse generic package build and uploaded by
build-components
. - If the package is retrieved successfully, its content is placed in the right folder (i.e.
tmp/tests/gitlab-workhorse
), preventing the building of the binaries whenscripts/setup-test-env
is run later on. - If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of
scripts/setup-test-env
- First download the GitLab Workhorse generic package build and uploaded by
Notes:
- The diff to achieve this is quite small, and we can see that
TestEnv
didn't have to be touched at all, so this is really an optimization that sits on top of what we have today (i.e. caching of GitLab Workhorse binaries). - The version of the package is the
workhorse
tree SHA (i.e.git rev-parse HEAD:workhorse
). - Since the
workhorse
shouldn't change too often, most of the MRs should be able to take advantage of the pre-built package, which would be built every two hours. - As a further iteration, the package could be downloaded by
TestEnv
directly, so that local GDK instances, and GitPod instances (not sure about that) would benefit from it as well (though, different architectures should be taken in account, see gitlab-development-kit#1391).
Improvements
scripts/setup-test-env
Duration of With package | Duration |
---|---|
No | 92s, 84s, 82s |
Yes | 42s, 44s, 46s |
It looks like we could already save 42 seconds on average for the setup-test-env
job.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR. -
Documentation is up-to-date: #353078 (closed) => !81289 (merged)
Related to #351035 (closed).
Merge request reports
Activity
changed milestone to %14.8
added Engineering Productivity [deprecated] Accepting merge requests maintenanceworkflow priority2 tooling (archive) + 1 deleted label
assigned to @rymai
1 Warning b48936ac: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. Pipeline Changes
This merge request contains changes to the pipeline configuration for the GitLab project.
Please consider the effect of the changes in this merge request on the following:
- Effects on different pipeline types
- Effects on non-canonical projects:
gitlab-foss
security
dev
- personal forks
- Effects on pipeline performance
Please consider communicating these changes to the broader team following the communication guideline for pipeline changes
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer ~"type::tooling" for CI, Danger Alex Buijs ( @alexbuijs
) (UTC+1, same timezone as@rymai
)Kerri Miller ( @kerrizor
) (UTC-8, 9 hours behind@rymai
)To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
DangerAllure report
allure-report-publisher
generated test report for b48936ac!review-qa-reliable:
test report
review-qa-smoke: test report
package-and-qa: test reportremoved [deprecated] Accepting merge requests label
added 1 commit
- f7eae216 - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- 364c6c7c - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- 3e5ffc79 - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- 046cfe61 - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- 213710f2 - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- e2d076ad - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- 06779441 - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- 3c6d24ac - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- e895a4f0 - ci: Build workhorse binaries and upload/download as generic package
This is very exciting, ace work @rymai
- Resolved by Lin Jen-Shin
So, it seems this could actually reduce the pipelines duration a bit (42 seconds), which is a nice unexpected side-effect improvement.
@ashmckenzie Could you please review? This is really a first iteration that focuses on the CI for now. Next step would be looking at how this could benefit GDK & GitPod. Thanks!
requested review from @ashmckenzie
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
removed review request for @ashmckenzie
added 481 commits
-
e895a4f0...4fd566f3 - 479 commits from branch
master
- 6abb01ee - ci: Build workhorse binaries and upload/download as generic package
- dfdb3e47 - ci: Use curl for downloading Workhorse binaries archive and use upx
-
e895a4f0...4fd566f3 - 479 commits from branch
added 1236 commits
-
dfdb3e47...ab524a54 - 1234 commits from branch
master
- 8bba0f5c - ci: Build workhorse binaries and upload/download as generic package
- 8a25bb8c - ci: Use curl for downloading Workhorse binaries archive and use upx
-
dfdb3e47...ab524a54 - 1234 commits from branch
added 1 commit
- 191ff796 - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- 54faf483 - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- 0b47b355 - ci: Use curl for downloading Workhorse binaries archive and use upx
added 58 commits
-
0b47b355...4333e7a1 - 56 commits from branch
master
- 3619d6b9 - ci: Build workhorse binaries and upload/download as generic package
- 33a88e0b - ci: Use curl for downloading Workhorse binaries archive and use upx
-
0b47b355...4333e7a1 - 56 commits from branch
added 1 commit
- 5284de9e - ci: Use curl for downloading Workhorse binaries archive and use upx
added 44 commits
-
5284de9e...43298757 - 42 commits from branch
master
- 3c17e0e3 - ci: Build workhorse binaries and upload/download as generic package
- e2298bcf - ci: Use curl for downloading Workhorse binaries archive and use upx
-
5284de9e...43298757 - 42 commits from branch
added 1 commit
- 9926a51b - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- 6cc5814a - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- b99f8129 - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- 9f35aca7 - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- 4ff832c2 - ci: Use curl for downloading Workhorse binaries archive and use upx
added 128 commits
-
4ff832c2...4d9be902 - 126 commits from branch
master
- f5c1146c - ci: Build workhorse binaries and upload/download as generic package
- 6954abca - ci: Use curl for downloading Workhorse binaries archive and use upx
-
4ff832c2...4d9be902 - 126 commits from branch
added 1 commit
- 2526d6ca - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- 4fda761a - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- 3a4a54c8 - ci: Use curl for downloading Workhorse binaries archive and use upx
added 1 commit
- ca3a9c3e - ci: Use curl for downloading Workhorse binaries archive and use upx
added 67 commits
-
ca3a9c3e...6dfa934b - 65 commits from branch
master
- 79a32b5c - ci: Build workhorse binaries and upload/download as generic package
- 90f1f60a - ci: Use curl for downloading Workhorse binaries archive and use upx
-
ca3a9c3e...6dfa934b - 65 commits from branch
added 1 commit
- c986de6b - ci: Use curl for downloading Workhorse binaries archive and use upx
added 15 commits
-
c986de6b...186d5e70 - 14 commits from branch
master
- fbcd2d5d - ci: Build workhorse binaries and upload/download as generic package
-
c986de6b...186d5e70 - 14 commits from branch
changed milestone to %14.9
added 1 commit
- 298c97e4 - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- bdbf5fed - ci: Build workhorse binaries and upload/download as generic package
added 1 commit
- 59b13cb5 - ci: Build workhorse binaries and upload/download as generic package
requested review from @ashmckenzie
requested review from @godfat-gitlab and removed review request for @ashmckenzie
@ashmckenzie
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline has been started.
For more info, please refer to the following links:
added 138 commits
-
59b13cb5...f2c2b556 - 137 commits from branch
master
- 57047163 - ci: Build workhorse binaries and upload/download as generic package
-
59b13cb5...f2c2b556 - 137 commits from branch
mentioned in issue #353078 (closed)
- Resolved by Lin Jen-Shin
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Lin Jen-Shin
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
- Resolved by Rémy Coutable
removed review request for @godfat-gitlab
added 71 commits
-
57047163...01756fc7 - 70 commits from branch
master
- dd872329 - ci: Build workhorse binaries and upload/download as generic package
-
57047163...01756fc7 - 70 commits from branch
enabled an automatic merge when the pipeline for ee81aa9d succeeds
- Resolved by Rémy Coutable
@godfat-gitlab It looks the latest pipeline failed. Do you want to re-enable MWPS?
assigned to @godfat-gitlab and unassigned @rymai
added 377 commits
-
dd872329...3dc4fec0 - 376 commits from branch
master
- b48936ac - ci: Build workhorse binaries and upload/download as generic package
-
dd872329...3dc4fec0 - 376 commits from branch
The
package-and-qa
job from pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/476075494 triggered https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines/476084548 downstream.The
gitlab-qa
downstream pipeline failed! .The
package-and-qa
job from pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/478808630 triggered https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines/478828329 downstream.The
gitlab-qa
downstream pipeline failed! .
requested review from @godfat-gitlab
assigned to @rymai
marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed
mentioned in merge request !81289 (merged)
@rymai Sorry, only came by this now
Triggering a new pipeline and setting Merge when pipeline succeeds.enabled an automatic merge when the pipeline for 18d85199 succeeds
mentioned in commit 9c103767
added workflowstaging-canary label
added workflowstaging label and removed workflowstaging-canary label
- Resolved by Rémy Coutable
mentioned in issue #353821 (closed)
added workflowcanary label and removed workflowstaging label
added workflowproduction label and removed workflowcanary label
added typemaintenance label and removed tooling (archive) label
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
mentioned in issue #357096 (closed)
mentioned in issue #371244 (closed)
mentioned in merge request !96297 (merged)