Skip to content
Snippets Groups Projects

ci: Build workhorse binaries and upload/download as generic package

All threads resolved!

What does this MR do and why?

This MR does two things:

  1. Adds a new build-components job that runs automatically for all GitLab.com gitlab-org/gitlab scheduled pipelines, for any master commit that touches the workhorse/ folder, and is manual for GitLab.com's gitlab-org's MRs.
    1. This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab's test suite (under tmp/tests/gitlab-workhorse).
    2. If the package URL returns a 404:
      1. It runs scripts/setup-test-env, so that the GitLab Workhorse binaries are built.
      2. It then creates an archive which contains the binaries and upload it as a generic package to https://gitlab.com/gitlab-org/gitlab/-/packages/.
  2. Change the setup-test-env job to:
    1. First download the GitLab Workhorse generic package build and uploaded by build-components.
    2. If the package is retrieved successfully, its content is placed in the right folder (i.e. tmp/tests/gitlab-workhorse), preventing the building of the binaries when scripts/setup-test-env is run later on.
    3. If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of scripts/setup-test-env

Notes:

  • The diff to achieve this is quite small, and we can see that TestEnv didn't have to be touched at all, so this is really an optimization that sits on top of what we have today (i.e. caching of GitLab Workhorse binaries).
  • The version of the package is the workhorse tree SHA (i.e. git rev-parse HEAD:workhorse).
  • Since the workhorse shouldn't change too often, most of the MRs should be able to take advantage of the pre-built package, which would be built every two hours.
  • As a further iteration, the package could be downloaded by TestEnv directly, so that local GDK instances, and GitPod instances (not sure about that) would benefit from it as well (though, different architectures should be taken in account, see gitlab-development-kit#1391).

Improvements

Duration of scripts/setup-test-env

With package Duration
No 92s, 84s, 82s
Yes 42s, 44s, 46s

It looks like we could already save 42 seconds on average for the setup-test-env job. :rocket:

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #351035 (closed).

Edited by Rémy Coutable

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Ash McKenzie
  • Ash McKenzie removed review request for @ashmckenzie

    removed review request for @ashmckenzie

  • Rémy Coutable added 481 commits

    added 481 commits

    • e895a4f0...4fd566f3 - 479 commits from branch master
    • 6abb01ee - ci: Build workhorse binaries and upload/download as generic package
    • dfdb3e47 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1236 commits

    added 1236 commits

    • dfdb3e47...ab524a54 - 1234 commits from branch master
    • 8bba0f5c - ci: Build workhorse binaries and upload/download as generic package
    • 8a25bb8c - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 191ff796 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 54faf483 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 0b47b355 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 58 commits

    added 58 commits

    • 0b47b355...4333e7a1 - 56 commits from branch master
    • 3619d6b9 - ci: Build workhorse binaries and upload/download as generic package
    • 33a88e0b - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 5284de9e - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 44 commits

    added 44 commits

    • 5284de9e...43298757 - 42 commits from branch master
    • 3c17e0e3 - ci: Build workhorse binaries and upload/download as generic package
    • e2298bcf - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 9926a51b - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 6cc5814a - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • b99f8129 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 9f35aca7 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 4ff832c2 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 128 commits

    added 128 commits

    • 4ff832c2...4d9be902 - 126 commits from branch master
    • f5c1146c - ci: Build workhorse binaries and upload/download as generic package
    • 6954abca - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 2526d6ca - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 4fda761a - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 3a4a54c8 - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • ca3a9c3e - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 67 commits

    added 67 commits

    • ca3a9c3e...6dfa934b - 65 commits from branch master
    • 79a32b5c - ci: Build workhorse binaries and upload/download as generic package
    • 90f1f60a - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • c986de6b - ci: Use curl for downloading Workhorse binaries archive and use upx

    Compare with previous version

  • Rémy Coutable added 15 commits

    added 15 commits

    Compare with previous version

  • Rémy Coutable changed milestone to %14.9

    changed milestone to %14.9

  • Rémy Coutable added 1 commit

    added 1 commit

    • 298c97e4 - ci: Build workhorse binaries and upload/download as generic package

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • bdbf5fed - ci: Build workhorse binaries and upload/download as generic package

    Compare with previous version

  • Rémy Coutable added 1 commit

    added 1 commit

    • 59b13cb5 - ci: Build workhorse binaries and upload/download as generic package

    Compare with previous version

  • Ash McKenzie requested review from @ashmckenzie

    requested review from @ashmckenzie

  • Ash McKenzie approved this merge request

    approved this merge request

  • Ash McKenzie requested review from @godfat-gitlab and removed review request for @ashmckenzie

    requested review from @godfat-gitlab and removed review request for @ashmckenzie

  • :wave: @ashmckenzie, thanks for approving this merge request.

    This is the first time the merge request is approved. To ensure full test coverage, a new pipeline has been started.

    For more info, please refer to the following links:

  • Rémy Coutable added 138 commits

    added 138 commits

    Compare with previous version

  • Rémy Coutable changed the description

    changed the description

  • mentioned in issue #353078 (closed)

  • Rémy Coutable changed the description

    changed the description

  • Lin Jen-Shin
  • Lin Jen-Shin
  • Lin Jen-Shin
  • Lin Jen-Shin removed review request for @godfat-gitlab

    removed review request for @godfat-gitlab

  • Rémy Coutable added 71 commits

    added 71 commits

    Compare with previous version

  • Lin Jen-Shin approved this merge request

    approved this merge request

  • Lin Jen-Shin resolved all threads

    resolved all threads

  • Lin Jen-Shin enabled an automatic merge when the pipeline for ee81aa9d succeeds

    enabled an automatic merge when the pipeline for ee81aa9d succeeds

  • Rémy Coutable assigned to @godfat-gitlab and unassigned @rymai

    assigned to @godfat-gitlab and unassigned @rymai

  • Rémy Coutable aborted the automatic merge because source branch was updated

    aborted the automatic merge because source branch was updated

  • Rémy Coutable added 377 commits

    added 377 commits

    Compare with previous version

  • requested review from @godfat-gitlab

  • assigned to @rymai

  • Rémy Coutable resolved all threads

    resolved all threads

  • Rémy Coutable marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

    marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

  • Rémy Coutable mentioned in merge request !81289 (merged)

    mentioned in merge request !81289 (merged)

  • @rymai Sorry, only came by this now :pray: Triggering a new pipeline and setting Merge when pipeline succeeds.

  • Lin Jen-Shin enabled an automatic merge when the pipeline for 18d85199 succeeds

    enabled an automatic merge when the pipeline for 18d85199 succeeds

  • I'll trigger the QA when it's ready.

  • merged

  • Lin Jen-Shin mentioned in commit 9c103767

    mentioned in commit 9c103767

  • Rémy Coutable
  • Rémy Coutable resolved all threads

    resolved all threads

  • mentioned in issue #353821 (closed)

  • added workflowcanary label and removed workflowstaging label

  • added workflowproduction label and removed workflowcanary label

  • added typemaintenance label and removed tooling (archive) label

  • mentioned in issue #357096 (closed)

  • mentioned in issue #371244 (closed)

  • Rémy Coutable mentioned in merge request !96297 (merged)

    mentioned in merge request !96297 (merged)

  • Please register or sign in to reply
    Loading