Experiment Validated: Transparent GitLab CI support for amd64 containers and builds on arm64 hardware

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

• Experiment summary
• Hypothesis
• Business problem
• Expected outcome
• Experiment design & implementation
• Known Concerns, Further Testing, Further Information Gathering

Experiment summary

We believe that GitLab and customers can configure arm runners to transparently (by default) run amd64 containers which would allow AutoDevOps, Secrets Management and other containers to NOT have to be built as Multi-Arch.

Furthermore (sorry a lot of value in this one), customers and GitLab can take advantage of much cheaper and faster arm hardware even when target production container builds are for amd64 (x86_64).

Hypothesis

We believe that GitLab and customers can configure arm runners to transparently (by default) run amd64 containers which would allow AutoDevOps, Secrets Management and other containers to NOT have to be built as Multi-Arch.

Furthermore (sorry a lot of value in this one), customers and GitLab can take advantage of much cheaper and faster arm hardware even when target production container builds are for amd64 (x86_64).

Business problem

For GitLab - customers are requesting that all GitLab provided containers be built as multi-arch so they can fully transition their runner fleets to Arm. This includes not only all of AutoDevOps, but any utility functionality like Secrets Management and Releases that rely on GitLab Maintained containers. Every additional container based feature would also need to be created as Multi-arch.

For Customers - CI compute bills can be very high due to the highly repetitious nature of CI. Cloud providers who have Arm technology give both higher performance and lower cost when Arm is utilitized. For instance Graviton4 provides up to 30% better compute performance, 50% more cores, and 75% more memory bandwidth than current generation Graviton3 processors. That is on top of similar compounding numbers for Graviton 3 over Graviton 2, Graviton 2 over the initial release and Graviton over x86.

Expected outcome

To verify that, we will:

• Setup a GitLab docker runner on arm64 hardware
• with docker buildx (via QEMU) support configured
• with the DOCKER_DEFAULT_PLATFORM set to 'linux/amd64' (in the runner config.toml as it will be the most universal across all distros when placed here) example (docs: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section):

  [[runners]]
    ...snip...
    environment = ["DOCKER_DEFAULT_PLATFORM=linux/amd64"]

And we’ll measure:

• Job output when .gitlab-ci.yml job does NOT set image:docker:platform
  • Desired result - runner runs the container as linux/amd64 due to the environment variable
• Job output when .gitlab-ci.yml job IS set to image:docker:platform: arm64 AND config.toml DOCKER_DEFAULT_PLATFORM=linux/amd64 (stays the same)
  • Desire Result - that specific runner job overrides the default for explicit use of arm.

Experiment design & implementation

The experiment was a success. The best outcomes are realized with no additional product changes. An Arm hardware machine runner docker can be defaulted to do amd64 (x86_64) builds when gitlab-ci.yml does not specify image:docker:platform AND the same runner can have individual jobs configured to use arm explicitly by setting image:docker:platform: linux/arm64

These screenshots show the evidence:

Known Concerns, Further Testing, Further Information Gathering

• Runner helper may run as hardware native container type?
• Cross build / emulation errors may be obscure to do root cause analysis on.
• Need to run tests on Auto DevOps and other utility containers to get a feel for compatibility. (Even if there are some incompatibilities, it may still be a lot cheaper to make them work under emulation than to move everything to 'maintained' multi-arch builds.)
  • counter-point - test whether GitLab CI containers that test good for emulation, could have their image:docker:platform forced to amd64.
• Talk to Arm company about prevalence and nature of failure cases.