Move exported helper images into separate package
This MR addresses #28014 (closed) by moving the exported helper images into a separate package, and no longer including them in the runner deb/rpm packages. This change excludes the fips runner package; we continue to include the single exported fips helper image in that package since you really want to use the fips helper if you're using the fips runner.
We want to do this because the current approach of including the exported images in every single runner deb/rpm package produces hughe packages (~509MiB) AND unnecessarily duplicates A LOT of data.
How much data? Note that this analysis excludes the fips images/packages since they have not changed!
For context consider that we build runner packages for 9 architectures and 68 OS/version combinations (35 deb and 33 rpm):
List of runner arch and OS/versions for which we create packages
archs
- x86_64
- i386
- aarch64
- arm64
- arm
- armhf
- riscv64
- s390x
- ppc64le
deb OS/versions
- ubuntu/oracular
- ubuntu/noble
- ubuntu/mantic
- ubuntu/lunar
- ubuntu/kinetic
- ubuntu/jammy
- ubuntu/impish
- ubuntu/focal
- ubuntu/bionic
- ubuntu/xenial
- debian/forky
- debian/trixie
- debian/bookworm
- debian/bullseye
- debian/buster
- debian/stretch
- linuxmint/wilma
- linuxmint/virginia
- linuxmint/victoria
- linuxmint/vera
- linuxmint/vanessa
- linuxmint/una
- linuxmint/uma
- linuxmint/ulyssa
- linuxmint/ulyana
- linuxmint/sonya
- linuxmint/serena
- linuxmint/sarah
- raspbian/forky
- raspbian/trixie
- raspbian/bookworm
- raspbian/bullseye
- raspbian/buster
- raspbian/stretch
- raspbian/jessie
rpm OS/versons
- el/9
- el/8
- el/7
- fedora/42
- fedora/41
- fedora/40
- fedora/39
- fedora/38
- fedora/37
- fedora/36
- fedora/35
- fedora/34
- fedora/33
- fedora/32
- ol/9
- ol/8
- ol/7
- ol/6
- opensuse/42.3
- sles/15.6
- sles/15.5
- sles/15.4
- sles/15.3
- sles/15.2
- sles/15.1
- sles/15.0
- sles/12.5
- sles/12.4
- sles/12.3
- amazon/2025
- amazon/2023
- amazon/2022
- amazon/2
Current runner deb/rpm packages are ~509M MiB. E.g.:
> ll out/deb out/rpm/
out/deb:
.rw-r--r-- 509Mi avb avb 27 Nov 10:54 gitlab-runner_amd64.deb
out/rpm/:
.rw-r--r-- 509Mi avb avb 27 Nov 10:53 gitlab-runner_amd64.rpmThis is a problem on it's own in that uploading those packages to packagecloud often times out because it takes so long.
For each runner release, we create and upload ~304GiB of data to packagecloud!:
509MiB * 9 (archs) * 68 (os/versions) = 311508MiB ~= 304GiBMost of that is the same data since all the runner packages include the exact same exported helper images!
After this change, we have the following:
> ll out/deb out/rpm/
out/deb:
.rw-r--r-- 485Mi avb avb 27 Nov 10:23 gitlab-runner-helper-images.deb
.rw-r--r-- 24Mi avb avb 26 Nov 15:13 gitlab-runner_amd64.deb
out/rpm/:
.rw-r--r-- 485Mi avb avb 27 Nov 10:25 gitlab-runner-helper-images.rpm
.rw-r--r-- 24Mi avb avb 26 Nov 15:13 gitlab-runner_amd64.rpmThe runner packages are reduced to ~24MiB, and we add one 485MiB package for each os/version. This results in ~47GiB of data to upload to packagecloud:
((24MiB * 9) + 485MiB) * 68 = 47668MiB ~= 47GiBThat's a savings of 304GiB - 47GiB = 257GiB in storage and bandwidth costs, plus however much CI time. Yo!
Changes to the mage targets are like so:
We've added to targets:
package:helperDeb creates a deb package with the exported runner-helper images
package:helperRpm creates an rpm package with the exported runner-helper imagesThe existing mage package:(deb|rpm)* targets have changed from this e.g.
> mage package:rpm64
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| TARGET INFO | | |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| Dependency | Type | Exists |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| fpm | OS binary | exec: "fpm": executable file not found in $PATH | ...ignore this
| out/binaries/gitlab-runner-linux-amd64 | File | Yes |
| out/helper-images/prebuilt-alpine-arm.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-arm64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-edge-riscv64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-s390x.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-x86_64-pwsh.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-x86_64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-arm.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-arm64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-ppc64le.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-s390x.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-x86_64-pwsh.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-x86_64.tar.xz | Docker image archive | Yes |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| Artifact | Type | Exists |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| out/rpm/gitlab-runner_amd64.rpm | File | Yes |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| Environment variable | Is set | Is default |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| GPG_KEYID | No | Yes |
| GPG_PASSPHRASE | No | Yes |
| PACKAGES_ITERATION | No | Yes |
+-------------------------------------------------------+----------------------+-------------------------------------------------+to this
> mage package:rpm64
+----------------------------------------+-----------+-------------------------------------------------+
| TARGET INFO | | |
+----------------------------------------+-----------+-------------------------------------------------+
| Dependency | Type | Exists |
+----------------------------------------+-----------+-------------------------------------------------+
| fpm | OS binary | exec: "fpm": executable file not found in $PATH |
| out/binaries/gitlab-runner-linux-amd64 | File | Yes |
+----------------------------------------+-----------+-------------------------------------------------+
| Artifact | Type | Exists |
+----------------------------------------+-----------+-------------------------------------------------+
| out/rpm/gitlab-runner_amd64.rpm | File | Yes |
+----------------------------------------+-----------+-------------------------------------------------+
| Environment variable | Is set | Is default |
+----------------------------------------+-----------+-------------------------------------------------+
| GPG_KEYID | No | Yes |
| GPG_PASSPHRASE | No | Yes |
| PACKAGES_ITERATION | No | Yes |
+----------------------------------------+-----------+-------------------------------------------------+And this is what the new targets look like.
> mage package:helperRpm
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| TARGET INFO | | |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| Dependency | Type | Exists |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| fpm | OS binary | exec: "fpm": executable file not found in $PATH |
| out/helper-images/prebuilt-alpine-arm.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-arm64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-edge-riscv64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-s390x.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-x86_64-pwsh.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-alpine-x86_64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-arm.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-arm64.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-ppc64le.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-s390x.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-x86_64-pwsh.tar.xz | Docker image archive | Yes |
| out/helper-images/prebuilt-ubuntu-x86_64.tar.xz | Docker image archive | Yes |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| Artifact | Type | Exists |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| out/rpm/gitlab-runner-helper-images.rpm | File | Yes |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| Environment variable | Is set | Is default |
+-------------------------------------------------------+----------------------+-------------------------------------------------+
| GPG_KEYID | No | Yes |
| GPG_PASSPHRASE | No | Yes |
| PACKAGES_ITERATION | No | Yes |
+-------------------------------------------------------+----------------------+-------------------------------------------------+Installing runner now looks like this. Note that this example installs runner and the helper-images from local packages, which does not resolve dependencies which aren't available in the package manager repo. In normal use (adding the runner packagecloud repo), the gitlab-runner-helper-images dependency would be auto-resolved and the gitlab-runner-helper-images package would be installed when installing the runner package.
Installing runner
> cat /etc/issue
Ubuntu 24.04.1 LTS \n \l
> ls -hl
total 510M
-rw-r--r-- 1 root root 485M Nov 27 19:02 gitlab-runner-helper-images.deb
-rw-r--r-- 1 root root 25M Nov 27 17:58 gitlab-runner_amd64.deb
> apt-get install ./gitlab-runner_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'gitlab-runner' instead of './gitlab-runner_amd64.deb'
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
gitlab-runner : Depends: gitlab-runner-helper-images but it is not installable
E: Unable to correct problems, you have held broken packages.
> apt-get install ./gitlab-runner-helper-images.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'gitlab-runner-helper-images' instead of './gitlab-runner-helper-images.deb'
The following NEW packages will be installed:
gitlab-runner-helper-images
0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 0 B/508 MB of archives.
After this operation, 506 MB of additional disk space will be used.
Get:1 /root/deb/gitlab-runner-helper-images.deb gitlab-runner-helper-images all 17.6.0~pre.237.g4c57a17a-1 [508 MB]
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package gitlab-runner-helper-images.
(Reading database ... 4379 files and directories currently installed.)
Preparing to unpack .../gitlab-runner-helper-images.deb ...
Unpacking gitlab-runner-helper-images (17.6.0~pre.237.g4c57a17a-1) ...
Setting up gitlab-runner-helper-images (17.6.0~pre.237.g4c57a17a-1) ...
> apt search gitlab-runner
Sorting... Done
Full Text Search... Done
gitlab-runner-helper-images/now 17.6.0~pre.237.g4c57a17a-1 all [installed,local]
GitLab Runner Helper Docker Images
> ls -lh /usr/lib/gitlab-runner/helper-images/
total 483M
-rw-r--r-- 1 root root 19M Nov 20 10:09 prebuilt-alpine-arm.tar.xz
-rw-r--r-- 1 root root 19M Nov 20 10:09 prebuilt-alpine-arm64.tar.xz
-rw-r--r-- 1 root root 21M Nov 20 10:09 prebuilt-alpine-edge-riscv64.tar.xz
-rw-r--r-- 1 root root 19M Nov 20 10:09 prebuilt-alpine-s390x.tar.xz
-rw-r--r-- 1 root root 74M Nov 20 10:09 prebuilt-alpine-x86_64-pwsh.tar.xz
-rw-r--r-- 1 root root 21M Nov 20 10:09 prebuilt-alpine-x86_64.tar.xz
-rw-r--r-- 1 root root 39M Nov 20 10:09 prebuilt-ubuntu-arm.tar.xz
-rw-r--r-- 1 root root 40M Nov 20 10:09 prebuilt-ubuntu-arm64.tar.xz
-rw-r--r-- 1 root root 45M Nov 20 10:09 prebuilt-ubuntu-ppc64le.tar.xz
-rw-r--r-- 1 root root 41M Nov 20 10:09 prebuilt-ubuntu-s390x.tar.xz
-rw-r--r-- 1 root root 105M Nov 20 10:09 prebuilt-ubuntu-x86_64-pwsh.tar.xz
-rw-r--r-- 1 root root 45M Nov 20 10:09 prebuilt-ubuntu-x86_64.tar.xz
> apt-get install ./gitlab-runner_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'gitlab-runner' instead of './gitlab-runner_amd64.deb'
The following additional packages will be installed:
adduser ca-certificates curl git git-man krb5-locales less libbrotli1 libbsd0 libcbor0.10 libcurl3t64-gnutls libcurl4t64 libedit2 liberror-perl libexpat1 libfido2-1
libgdbm-compat4t64 libgdbm6t64 libgssapi-krb5-2 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-common libldap2 libnghttp2-14 libperl5.38t64 libpsl5t64
librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 netbase openssh-client openssl patch
perl perl-modules-5.38 publicsuffix xauth
Suggested packages:
liblocale-gettext-perl cron quota ecryptfs-utils gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn
docker-engine gdbm-l10n krb5-doc krb5-user libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql
keychain libpam-ssh monkeysphere ssh-askpass ed diffutils-doc perl-doc libterm-readline-gnu-perl | libterm-readline-perl-perl make libtap-harness-archive-perl
The following NEW packages will be installed:
adduser ca-certificates curl git git-man gitlab-runner krb5-locales less libbrotli1 libbsd0 libcbor0.10 libcurl3t64-gnutls libcurl4t64 libedit2 liberror-perl libexpat1
libfido2-1 libgdbm-compat4t64 libgdbm6t64 libgssapi-krb5-2 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-common libldap2 libnghttp2-14 libperl5.38t64
libpsl5t64 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh-4 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 netbase openssh-client
openssl patch perl perl-modules-5.38 publicsuffix xauth
0 upgraded, 49 newly installed, 0 to remove and 6 not upgraded.
Need to get 19.5 MB/45.2 MB of archives.
After this operation, 199 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
...
Setting up gitlab-runner (17.6.0~pre.236.g32ad4755-1) ...
GitLab Runner: creating gitlab-runner...
Home directory skeleton not used
INFO[0000] Binary was compiled with FIPS mode, but an external SSL library was not enabled.
Runtime platform arch=amd64 os=linux pid=2580 revision=a45b5bfd version=17.6.0~pre.233.ga45b5bfd
gitlab-runner: exit status 1
INFO[0000] Binary was compiled with FIPS mode, but an external SSL library was not enabled.
Runtime platform arch=amd64 os=linux pid=2600 revision=a45b5bfd version=17.6.0~pre.233.ga45b5bfd
gitlab-ci-multi-runner: exit status 1
INFO[0000] Binary was compiled with FIPS mode, but an external SSL library was not enabled.
Runtime platform arch=amd64 os=linux pid=2658 revision=a45b5bfd version=17.6.0~pre.233.ga45b5bfd
INFO[0000] Binary was compiled with FIPS mode, but an external SSL library was not enabled.
Runtime platform arch=amd64 os=linux pid=2677 revision=a45b5bfd version=17.6.0~pre.233.ga45b5bfd
INFO: Docker installation not found, skipping clear-docker-cache
Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
Processing triggers for ca-certificates (20240203) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
> apt search gitlab-runner
Sorting... Done
Full Text Search... Done
gitlab-runner/now 17.6.0~pre.236.g32ad4755-1 amd64 [installed,local]
GitLab Runner
gitlab-runner-helper-images/now 17.6.0~pre.237.g4c57a17a-1 all [installed,local]
GitLab Runner Helper Docker ImagesEdited by Axel von Bertoldi