Skip to content

Generate packagecloud packages with Mage

Georgi N. Georgiev requested to merge generate-packagecloud-packages-with-mage into main

What does this MR do?

This MR is the first stepping stone for Runner release process should check the existen... (#36510).

It focuses on introducing Mage in the context of building packagecloud packages, replacing old make targets and adding a couple.

We have had multiple issues where some resource is not uploaded but we rely on users to report that.

In order to be able to verify that resources are indeed uploaded we need a single source of truth to report these resources. The issue is that we did not have that.

We used to have DEB_PLATFORMS and RPM_PLATFORMS in the Makefile but that went away with the introduction of the Go script for packagecloud. This also broke the make target for yanking packagecloud packages as these variables no longer existed.

We also had DEB_ARCHS and RPM_ARCHS but those were kept up-to-date manually and served virtually no purpose.

We had a lot of make targets in Makefile.packages.mk but exporting all the architectures and combinations of distributions and versions is a level of Makefile black magic which I am not comfortable nor capable of.

The solution is to start migrating scripts to Go but seriously. We have a few but they are disconnected from the general Make target and are rather more ad-hoc.

Adding Mage to the mix makes it easy to share variables and functions across the Go files making it so we have a single source of truth: !4323 (diffs).

From the packageBuilds variable we can go on to generate all the Mage targets for all the architectures and distributions. This allows us to also check that these targets have indeed been generated the same way we check for generated mocks. This reduces the chances that we will have a disconnect in what we define and what we build.

From there we can use these build definitions to:

  • Generate the final list of files that will be uploaded to packagecloud: mage package:filenames
  • List all architectures in the make version target while sharing the same source of the data
  • Generate mage targets with code generation reducing manual boilerplate, which we suffer from greatly in make
  • Use the existing packagecloud script in addition with the new Go code to yank all packagecloud packages

And in the near future:

  • Export all resources listed in #36510 in a job
  • Check all these resources in a separate pipeline thats part of the release process

Why was this MR needed?

What's the best way to test this MR?

Pipeline should pass. Running mage in the root dir should output:

Targets:
  generate                 runs go generate for all files in the magefiles directory
  package:archs            prints the list of architectures as they appear in the final package's filename for either "deb" or "rpm"
  package:deb              builds deb package
  package:deb32Bit         builds deb package for 386
  package:deb64Bit         builds deb package for amd64
  package:debArm32Bit      builds deb package for arm arm
  package:debArm64Bit      builds deb package for arm64 arm64
  package:debIbm           builds deb package for s390x ppc64le
  package:debRiscv64Bit    builds deb package for riscv64
  package:deps             makes sure the packages needed to build rpm and deb packages are available on the system
  package:filenames        prints the final names of the packages for all supported architectures for a version and a distribution
  package:generate         generates the Mage package build targets
  package:prepare          prepares the filesystem permissions for packages
  package:rpm              builds rpm package
  package:rpm32Bit         builds rpm package for 386
  package:rpm64Bit         builds rpm package for amd64
  package:rpmArm32Bit      builds rpm package for arm arm
  package:rpmArm64Bit      builds rpm package for arm64 arm64
  package:rpmFips          builds rpm package for fips
  package:rpmIbm           builds rpm package for s390x ppc64le
  package:rpmRiscv64Bit    builds rpm package for riscv64
  packageCloud:releases    prints the list of dist/release pairs supported by PackageCloud and not in their EOL stage
  packageCloud:yank        yanks all packages from PackageCloud for the specified version

What are the relevant issue numbers?

#36510

Edited by Georgi N. Georgiev

Merge request reports