Push runner linux packages to Pulp

Continuing with the pulp migration work, the MR adds the ability to push runner release packages (rpm and deb) to pulp.

Of course I started with the packagecloud equivalent as a reference. Pushing deb packages to pulp is fairly similar to pushing them to packagecloud in that each package must be pushed/uploaded to each distro/release combo, and the repo auto-detects the architecture from the package, so there's no special handling or any package arch. The code is conceptually similar, but I removed the retry-on-specific-failure mechanism since we added this specifically because the packagecloud push API/service is so flaky. If we do need this for pulp too, we can add it.

However, pushing rpm packages to pulp is very different from deb packages or packagecloud. For starters, with the rpm pulp repos, each distro/release/arch tuple gets its own repository which must be specified in the --repository argument of the push command. Handling of the gitlab-runner-helper-images package is different too (which has architecture noarch), in that there are no noarch repositories, and that package has to instead be pushed to every other distro/release/arch repository. This sucks because that file is ~500MB. @balasankarc presented one option here, which is to upload the package to one distro/release/arch repo, and then link it to all the other distro/release/arch repos. This is much more palatable, so this is what I've implemented. However, because we upload the same package/file to every distro/release for a given arch, this approach is also applicable to all other rpm packages, so I have applied it to pushing all rpm packages.

The net result of the above is that uploading rpm packages is done as follows:

For each package/file:

  1. Push/upload the package to any distro/package/arch pulp repo.
  2. Query that pulp repo to get pulp's internal url to the package. Annoyingly, there is no metadata returned by the command to push a package that can be used to identify that package in the "repo list" result. Instead, I have used information from the package itself (name, arch, version, revision) to identify the package. I think this is enough information to reliably identify the package.
  3. Link that package to all the other distro/package/arch. Note that for runner packages, there is only one arch (the one specified in the package), whereas for the gitlab-runner-helper-images package, the archs will be all the archs we support for rpm distros (currently x86_64 i686 aarch64 armhfp riscv64 loongarch64 s390x ppc64le).

I've tested this extensively in https://pulp.pre.gitlab.net/runner/, on both the stable and unstable release branches and am satisfied it works. If anyone else wants to test manually, here are the steps.

  1. Get a recent runner CI image with pulp installed (e.g docekr pull registry.gitlab.com/gitlab-org/ci-cd/runner-tools/base-images/ci:2271833038)
  2. run that image and mount the runner project folder somewhere, and cd to the runner folder (e.g. docker run -it --rm -v $PWD:/runner:Z registry.gitlab.com/gitlab-org/ci-cd/runner-tools/base-images/ci:2271833038 bash)
  3. build all the runner binaries with make runner-bin runner-bin-fips
  4. build the runner packages with mage package:deb32 package:deb64 package:debArm32 package:debArm64 package:debIbm package:debLoong64 package:debRiscv64 package:rpm32 package:rpm64 package:rpmArm32 package:rpmArm64 package:rpmFips package:rpmIbm package:rpmLoong64 package:rpmRiscv64. You'll need to export PACKAGES_ITERATION=1 for this to work.
  5. build the runner helper images package with mage package:helpersDeb package:helpersRpm. You will get warning about missing files like out/helper-images/prebuilt-alpine-arm.tar.xz. Just use touch on all those to create stub/dummy files.
  6. export 4 variables:
PULP_URL="https://pulp.pre.gitlab.net"
PULP_USER="runner"
PULP_PASSWORD="..."
PRIVATE_TOKEN='...'

Ask me for PULP_PASSWORD, and PRIVATE_TOKEN can by any of your private tokens

  1. upload some packages with mage pulp:push <deb|rpm> <stable|unstable> <distro>. You can see the supported distros with mage pulp:supportedOSVersions <deb|rpm> <stable|unstable>

I've included some unit tests too.

🚧 Best reviewed commit-at-a-time.

Closes https://gitlab.com/gitlab-org/gitlab-runner/-/issues/39120+

Edited by Axel von Bertoldi

Merge request reports

Loading