[Re-]Introduce CNG build stage images (!1422) · Merge requests · GitLab.org / Build / CNG

What does this MR do?

❗ This MR is a reboot of !1308 (merged). !1308 (merged) was reverted after it caused an issue with auto-deploy pipelines. This was discussed and addressed in !1389 (closed). This MR re-introduces the changes of !1308 (merged) and includes changes from !1389 (closed) as well.

The description is copied from !1308 (merged).

Overview

This MR contains:

core and go images for CNG build stages. core is a base build environment with essential build tools, such as make and gcc, development libraries and headers, and helper scripts. go is based on core builder and adds Go compiler to the mix.
BUILDER_PIPELINE which builds different flavors of core and go builder images and allows verifying them by triggering typical pipelines and passing their reference.

Some definitions

flavor i.e. Debian, UBI, FIPS
family is the category of the image, i.e. builders, assets, runtime (this is further expanded in gitlab-org/charts/gitlab#4217).
flavored Dockerfile is a Dockerfile that is suffixed with the flavor, e.g. Dockerfile.fips. When a flavored Dockerfile does not exist the default Dockerfile is used. For fips flavor, before falling back to the default Dockerfile the UBI-flavored Dockerfile is preferred.

CNG build stage images, aka "builder images"

core image contains:

All the essential build tools
All development headers and build dependencies that are required in other CNG Docker images
Utility scripts including scripts for downloading and verifying archives and artifacts.

go builder image is based on core image and adds Go language compiler and tools.

The core contains a few helper scripts which can be used in the build stage of other CNG images:

fetch: download and extract an archive file
gitlab-fetch: download and extract repository archive from GitLab
apply-patches: populates and applies the specified patches
cleanup-eggs: removes Python packages clutter
cleanup-gems: removes Ruby Gems clutter
detect-distro: detects Linux distribution, version, and architecture

Builder pipeline

The BUILDER_PIPELINE type is limited to the changes to builders/ directory. This directory is intended to group together the images that belong to the builder family.

This pipeline, which is supposed to only run on master and MR branches of .com repository, builds the different flavors of core and go builder images and add manual triggers for verifying the images. Each flavor has its own trigger. The triggers pass the reference the corresponding builder images of the flavor that are built for the current branch to the downstream pipelines.

Note that the build scripts of this pipeline are not added to build.sh and are written in a verbose style. This is done intentionally in order to allow further improvements to build.sh.

Related issues

Related to:

Further work

Add Rust as a builder image. This will allow replacing gitlab-rust image.
Use docker bake as a build tool to replace the mixture of shell script and CI configuration to imitate what a proper build tool does.

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

Merge Request Title, and Description are up to date, accurate, and descriptive
MR targeting the appropriate branch
MR has a green pipeline on GitLab.com
When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

Test plan indicating conditions for success has been posted and passes
Documentation created/updated
Integration tests added to GitLab QA
The impact any change in container size has should be evaluated

Edited Jul 25, 2023 by Hossein Pursultani

[Re-]Introduce CNG build stage images