ci: Build workhorse binaries and upload/download as generic package

What does this MR do and why?

This MR does two things:

1. Adds a new build-components job that runs automatically for all GitLab.com gitlab-org/gitlab scheduled pipelines, for any master commit that touches the workhorse/ folder, and is manual for GitLab.com's gitlab-org's MRs.
   1. This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab's test suite (under tmp/tests/gitlab-workhorse).
   2. If the package URL returns a 404:
      1. It runs scripts/setup-test-env, so that the GitLab Workhorse binaries are built.
      2. It then creates an archive which contains the binaries and upload it as a generic package to https://gitlab.com/gitlab-org/gitlab/-/packages/.
2. Change the setup-test-env job to:
   1. First download the GitLab Workhorse generic package build and uploaded by build-components.
   2. If the package is retrieved successfully, its content is placed in the right folder (i.e. tmp/tests/gitlab-workhorse), preventing the building of the binaries when scripts/setup-test-env is run later on.
   3. If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of scripts/setup-test-env

Notes:

• The diff to achieve this is quite small, and we can see that TestEnv didn't have to be touched at all, so this is really an optimization that sits on top of what we have today (i.e. caching of GitLab Workhorse binaries).
• The version of the package is the workhorse tree SHA (i.e. git rev-parse HEAD:workhorse).
• Since the workhorse shouldn't change too often, most of the MRs should be able to take advantage of the pre-built package, which would be built every two hours.
• As a further iteration, the package could be downloaded by TestEnv directly, so that local GDK instances, and GitPod instances (not sure about that) would benefit from it as well (though, different architectures should be taken in account, see gitlab-development-kit#1391).

Improvements

Duration of scripts/setup-test-env

With package	Duration
No	92s, 84s, 82s
Yes	42s, 44s, 46s

It looks like we could already save 42 seconds on average for the setup-test-env job.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.
• Documentation is up-to-date: #353078 (closed) => !81289 (merged)

Related to #351035 (closed).

.gitlab/ci/rails.gitlab-ci.yml

@@ -193,15 +193,13 @@ setup-test-env:
   stage: prepare
   variables:
     SETUP_DB: "false"
+    TMP_TEST_GITLAB_WORKHOSE_FOLDER: "${TMP_TEST_FOLDER}/gitlab-workhorse"
   script:
-    - workhorse_tree=$(git rev-parse HEAD:workhorse)
-    - workhorse_binaries_file="workhorse-${workhorse_tree}.tar.gz"
-    - 'wget --header="JOB-TOKEN: $CI_JOB_TOKEN" ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/workhorse/${workhorse_tree}/${workhorse_binaries_file} || true'
-    - if [ -f "${workhorse_binaries_file}" ]; then
-        mkdir -p tmp/tests/gitlab-workhorse
-        tar -xzvf ${workhorse_binaries_file} -C tmp/tests
-      fi
+    - source scripts/gitlab_workhorse_component_helpers.sh
+    - run_timed_command "download_gitlab_workhorse_package" || true
+    - run_timed_command "extract_gitlab_workhorse_package" || true
     - run_timed_command "scripts/setup-test-env"
+    - run_timed_command "filter_gitlab_workhorse_binaries"
     - echo -e "\e[0Ksection_start:`date +%s`:gitaly-test-build[collapsed=true]\r\e[0KCompiling Gitaly binaries"
     - run_timed_command "scripts/gitaly-test-build"  # Do not use 'bundle exec' here
     - echo -e "\e[0Ksection_end:`date +%s`:gitaly-test-build\r\e[0K"
@@ -209,28 +207,24 @@ setup-test-env:
     expire_in: 7d
     paths:
       - config/secrets.yml
-      - tmp/tests/gitaly/_build/bin/
-      - tmp/tests/gitaly/_build/deps/git/install
-      - tmp/tests/gitaly/config.toml
-      - tmp/tests/gitaly/gitaly2.config.toml
-      - tmp/tests/gitaly/internal/
-      - tmp/tests/gitaly/internal_gitaly2/
-      - tmp/tests/gitaly/internal_sockets/
-      - tmp/tests/gitaly/Makefile
-      - tmp/tests/gitaly/praefect.config.toml
-      - tmp/tests/gitaly/ruby/
-      - tmp/tests/gitlab-elasticsearch-indexer/bin/gitlab-elasticsearch-indexer
-      - tmp/tests/gitlab-shell/
-      - tmp/tests/gitlab-test-fork/
-      - tmp/tests/gitlab-test-fork_bare/
-      - tmp/tests/gitlab-test/
-      - tmp/tests/gitlab-workhorse/gitlab-zip-metadata
-      - tmp/tests/gitlab-workhorse/gitlab-zip-cat
-      - tmp/tests/gitlab-workhorse/gitlab-workhorse
-      - tmp/tests/gitlab-workhorse/gitlab-resize-image
-      - tmp/tests/gitlab-workhorse/WORKHORSE_TREE
-      - tmp/tests/repositories/
-      - tmp/tests/second_storage/
+      - ${TMP_TEST_FOLDER}/gitaly/_build/bin/
+      - ${TMP_TEST_FOLDER}/gitaly/_build/deps/git/install
+      - ${TMP_TEST_FOLDER}/gitaly/config.toml
+      - ${TMP_TEST_FOLDER}/gitaly/gitaly2.config.toml
+      - ${TMP_TEST_FOLDER}/gitaly/internal/
+      - ${TMP_TEST_FOLDER}/gitaly/internal_gitaly2/
+      - ${TMP_TEST_FOLDER}/gitaly/internal_sockets/
+      - ${TMP_TEST_FOLDER}/gitaly/Makefile
+      - ${TMP_TEST_FOLDER}/gitaly/praefect.config.toml
+      - ${TMP_TEST_FOLDER}/gitaly/ruby/
+      - ${TMP_TEST_FOLDER}/gitlab-elasticsearch-indexer/bin/gitlab-elasticsearch-indexer
+      - ${TMP_TEST_FOLDER}/gitlab-shell/
+      - ${TMP_TEST_FOLDER}/gitlab-test-fork/
+      - ${TMP_TEST_FOLDER}/gitlab-test-fork_bare/
+      - ${TMP_TEST_FOLDER}/gitlab-test/
+      - ${TMP_TEST_FOLDER}/repositories/
+      - ${TMP_TEST_FOLDER}/second_storage/
+      - ${TMP_TEST_GITLAB_WORKHOSE_FOLDER}/
     when: always
 
 build-components:
@@ -238,21 +232,18 @@ build-components:
     - setup-test-env
     - .rails:rules:build-components
   script:
+    - source scripts/gitlab_workhorse_component_helpers.sh
+    - run_timed_command "download_gitlab_workhorse_package" || true
+    # Early exit if the package already exists
+    - gitlab_workhorse_archive_doesnt_exist || exit 0
     - run_timed_command "scripts/setup-test-env"
-    - workhorse_tree=$(git rev-parse HEAD:workhorse)
-    - workhorse_binaries_file="workhorse-${workhorse_tree}.tar.gz"
-    - echo $workhorse_tree $workhorse_binaries_file
-    - tar -czvf ${workhorse_binaries_file} gitlab-resize-image gitlab-zip-cat gitlab-zip-metadata gitlab-workhorse WORKHORSE_TREE -C tmp/tests/gitlab-workhorse
-    - tar -tvf ${workhorse_binaries_file}
-    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file ${PACKAGENAME}.${VERSION}.tar.gz ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${workhorse}/${workhorse_tree}/${workhorse_binaries_file}'
+    - run_timed_command "filter_gitlab_workhorse_binaries"
+    - run_timed_command "create_gitlab_workhorse_archive"
+    - run_timed_command "upload_gitlab_workhorse_package"
   artifacts:
     expire_in: 7d
     paths:
-      - tmp/tests/gitlab-workhorse/gitlab-zip-metadata
-      - tmp/tests/gitlab-workhorse/gitlab-zip-cat
-      - tmp/tests/gitlab-workhorse/gitlab-workhorse
-      - tmp/tests/gitlab-workhorse/gitlab-resize-image
-      - tmp/tests/gitlab-workhorse/WORKHORSE_TREE
+      - ${TMP_TEST_GITLAB_WORKHOSE_FOLDER}/
 
 update-setup-test-env-cache:
   extends: