Provide docker images for alpine 3.12 (default), 3.13 and 3.14.
What does this MR do?
Due to https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27913 we need to provide an updated version of alpine to avoid security vulnerabilities.
We bumped to alpine 3.13 in !3057 (merged) as an intermediate step in order to avoid compability issues with older docker versions https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27913#note_637754156
We then reverted back to 3.12 since customers reported issues with 3.13 !3157 (merged)
This MR should optimise the pipeline a bit since helper images and helper images binaries are now split, some jobs that only wanted the binaries now don't have to wait for all the helper images as well etc. Also the helper images building and pushing are parallel now.
With this MR we will introduce the following versions for the GitLab Runner and GitLab Runner Helper docker images:
- gitlab-runner(-helper):alpine-{arch} (The default version is still alpine 3.12.0)
- gitlab-runner(-helper):alpine3.12-{arch}
- gitlab-runner(-helper):alpine3.13-{arch}
- gitlab-runner(-helper):alpine3.14-{arch}
Note:
The alpine 3.13 and 3.14 images are not included in the packages final deb and rpm since that would increase the size of the packages astronomically, instead they will be pulled from our registry or DockerHub depending on the user's preference. There have also been ideas to not include them at all so I think this approach makes sense.
Why was this MR needed?
To allow for users to use newer alpine versions without breaking behavior for old users.
What's the best way to test this MR?
-
Make sure that the correct images and versions are generated and built for the
helper images
anddevelopment docker
jobs -
To test building and publishing Runner images run:
PUBLISH_IMAGES=true PUSH_TO_DOCKER_HUB=true DOCKER_HUB_USER=<YOUR_USER> DOCKER_HUB_PASSWORD=<YOUR_PASSWORD> DOCKER_HUB_NAMESPACE=<YOUR_USER> IMAGE_FLAVOR=alpine3.13 make release_docker_images
For image flavor use one of alpine, alpine3.13, alpine3.14, ubuntu
- To test building and publishing Helper images run:
make helper-dockerarchive
This will build images for all platforms and architectures. Alternatively to build them separately run:
make "helper-dockerarchive-${IMAGE_BASE}"
Replace IMAGE_BASE with either alpine, alpine3.13, alpine3.14, ubuntu
To test the pushing and importing of the helper archives run the following commands:
PUBLISH_IMAGES=true PUSH_TO_DOCKER_HUB=true DOCKER_HUB_USER=<YOUR_USER> DOCKER_HUB_PASSWORD=<YOUR_PASSWORD> DOCKER_HUB_NAMESPACE=<YOUR_USER> IMAGE_FLAVOR=alpine3.12 IMAGE_TAG="" make release_helper_docker_images
PUBLISH_IMAGES=true PUSH_TO_DOCKER_HUB=true DOCKER_HUB_USER=<YOUR_USER> DOCKER_HUB_PASSWORD=<YOUR_PASSWORD> DOCKER_HUB_NAMESPACE=<YOUR_USER> IMAGE_FLAVOR=alpine3.12 IMAGE_TAG="alpine3.12-" make release_helper_docker_images
PUBLISH_IMAGES=true PUSH_TO_DOCKER_HUB=true DOCKER_HUB_USER=<YOUR_USER> DOCKER_HUB_PASSWORD=<YOUR_PASSWORD> DOCKER_HUB_NAMESPACE=<YOUR_USER> IMAGE_FLAVOR=alpine3.13 IMAGE_TAG="alpine3.13-" make release_helper_docker_images
PUBLISH_IMAGES=true PUSH_TO_DOCKER_HUB=true DOCKER_HUB_USER=<YOUR_USER> DOCKER_HUB_PASSWORD=<YOUR_PASSWORD> DOCKER_HUB_NAMESPACE=<YOUR_USER> IMAGE_FLAVOR=alpine3.14 IMAGE_TAG="alpine3.14-" make release_helper_docker_images
PUBLISH_IMAGES=true PUSH_TO_DOCKER_HUB=true DOCKER_HUB_USER=<YOUR_USER> DOCKER_HUB_PASSWORD=<YOUR_PASSWORD> DOCKER_HUB_NAMESPACE=<YOUR_USER> IMAGE_FLAVOR=ubuntu IMAGE_TAG="ubuntu-" make release_helper_docker_images
What are the relevant issue numbers?
Closes https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27913
Merge request reports
Activity
assigned to @ggeorgiev_gitlab
added [Deprecated] Category:Runner devopsverify grouprunner labels
added sectionops label
added 1 commit
- 1b22c06b - build runner images for multiple alpine versions
added 135 commits
-
45664233...0e764907 - 125 commits from branch
main
- f94f5d2b - Upgrade alpine version to 3.13
- f980f6ae - Update ALPINE_IMAGE_VERSION to 3.13
- 7b75ef9b - Helper image alpine3.13
- 8142d513 - Alpine3.13 for runner images
- 2a60c4da - build helper images for multiple alpine versions
- fee6cfad - fix build helper images for multiple alpine versions
- 9b3ca2af - default alpine helper is 3.12
- b1a8fa25 - add loggin
- 53769c84 - quick fix for tags
- 7c8d756b - release docker images with multiple flavor versions
Toggle commit list-
45664233...0e764907 - 125 commits from branch
added 1 commit
- 10612ccd - release helper images with multiple alpine versions
added 1 commit
- 468499cf - don't introduce the new alpine versions to packages
@ajwalker Can you review this?
I hope we can fit it into 14.5
If the MR doesn't provide enough context we can always do a quick zoom chat if you have any questions
@ggeorgiev_gitlab Looks like there's a couple of issues with the powershell docker release jobs
changed milestone to %14.5
requested review from @ajwalker
added maintenancepipelines label
added tooling (archive) label
added runner-distribution label and removed tooling (archive) label
added tooling (archive) label
added 59 commits
-
421b32ba...3bc85cf8 - 28 commits from branch
main
- aafa314e - Upgrade alpine version to 3.13
- 94433b59 - Update ALPINE_IMAGE_VERSION to 3.13
- e605d83d - Helper image alpine3.13
- a225c519 - Alpine3.13 for runner images
- 550b2442 - build helper images for multiple alpine versions
- eed70664 - fix build helper images for multiple alpine versions
- 17258732 - default alpine helper is 3.12
- 0fb2127f - add loggin
- 16754661 - quick fix for tags
- 479aa645 - release docker images with multiple flavor versions
- 22874ed7 - release helper images with multiple alpine versions
- fd8a5869 - add docs
- 787e4bd6 - fix helper make rules
- e079bc17 - fix k8s docs
- c7056e5b - don't introduce the new alpine versions to packages
- 051bdb22 - Don't add version to default alpine images
- 14d696e9 - Upgrade alpine to 3.14.2 in all tests and variables
- 514a9574 - Make helper images build parallel
- f6d1b909 - fix release_docker_images
- 70a8bd00 - fix helper-dockerarchive-ubuntu
- 66bc6208 - fix jobs artifacts and dependencies
- 0a8b6272 - add logging for building helper images
- 936e4522 - fix building multiple helper image versions
- 74fe5042 - fix CI image pwsh version
- a7fa92d0 - fix tagging and pushing images for Runner
- c6b99d23 - fix helper images and binaries dependencies
- c4694a86 - make alpine versions naming consistent
- 265435b2 - fix and optimise helper images pushing
- 1bb0bf5d - fix build yaml
- 23bdf0b6 - fix build yaml
- 264a96d2 - provide helper images to package jobs
Toggle commit list-
421b32ba...3bc85cf8 - 28 commits from branch
added 1 commit
- 359bdac4 - Improve docs and add a couple tests for alpine flavors
added 1 commit
- 46af4594 - Improve docs and add a couple tests for alpine flavors
@sselhorn Can you do docs review while I figure out why the Windows jobs fail?
@ajwalker Windows jobs are fixed, should be ready for your review as well
Hi @ggeorgiev_gitlab Two comments/suggestions for you. I'll click Approve so no need to assign back to me unless you have questions. Thanks!
Thanks! This looks good to me, so adding to merge train.
It feels like we keep adding and adding to our package building/releasing and it keeps getting more and more complex, despite doing mostly the same things. That's nothing against what you've done here, as you've just extended the support. But I do wonder if in general there's a cleaner solution. I've no idea what that looks like though. Maybe we've outgrown make and bash for this?
Thank you @ajwalker! This is how I felt during the development of this MR. The build process was extremely hard to follow and work with until I learned it by hearth. Certainly, as you mentioned the solution isn't to stop adding more things, e.g. not supporting more docker images are other projects, e.g. powershell support many more than we do. Maybe a refactor/reorgranisation/rewrite/reimagining of the pipeline and build scripts is in order in the future.
Also I am not sure whether make is exactly true, make is doing just fine, but it's at a point where people unfamiliar with its complicated features look at it and think its closer to actual magic than programming. But most certainly, I am not a fan of the make / bash combo for anything more complex. Every time I think about it Magefiles sound like a good option.
requested review from @sselhorn
added 1 commit
- 0afb9537 - set PWSH_VERSION only for ci image and windows images
added 1 commit
- 0916a97f - set PWSH_VERSION only for ci image and windows images
added Technical Writing docsfeature documentation labels
- Resolved by Georgi N. Georgiev
- Resolved by Georgi N. Georgiev
removed review request for @sselhorn
enabled automatic add to merge train when the pipeline for 0a06841f succeeds
mentioned in commit 9f74b3fc
mentioned in merge request !3193 (merged)
added security-awardsnomination label
mentioned in merge request !2969 (closed)
mentioned in issue #27754 (closed)
added typemaintenance label and removed tooling (archive) label
mentioned in merge request !4072 (merged)