Efforts to once again support amazon ECR as a docker registry for GitLab Runner's images
Timeline:
- A while back there was an idea to push only to our own registry. As per #29192 (comment 1181859546) I think this idea is scrapped.
- We stopped pushing to ECR from our pipelines temporarily - https://gitlab.com/gitlab-sirt/incident_2382/-/issues/1
- We opened https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29384 to track the effort of republishing to ECR.
- Meanwhile we were pushing the images manually using
scopeo
. Find the very crude fish scripts I was using bellow. - We then started pushing to ECR again with !3568 (merged)
- Shortly after we hit a limit on ECR repositories where images are capped to 10k - #29424 (closed)
- The MR above was reverted and we once again stopped pushing to ECR
- We requested from AWS to raise our limits from 10k to 20k as a temporary solution and also a good measure - https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29155#note_1171737236
- While waiting for the limits to be raised we started working on #29070 in order to garbage collect any dev images that are no longer used since even with 20k limit our ECR repositories were bound to get filled up again.
- I opened ggeorgiev_gitlab/runner-docker-images-registry-cleanup!1 (merged) so we could have an MVC which we can use to cleanup the images as soon as possible to unblock ECR pushing.
- As of the time of writing creating this issue I just deleted 7k images from Amazon ECR based on the criteria listed here ggeorgiev_gitlab/runner-docker-images-registry-cleanup!1 (diffs)
- All release images of GitLab Runner should be present in ECR as of 22.11.2022.
- As a next step we will start running ggeorgiev_gitlab/runner-docker-images-registry-cleanup!1 (merged) on a schedule to keep the amount of images in ECR in a manageable level (issue coming up, this bullet point will also be updated)
- The next immediate step will be to merge !3743 so images are once again automatically pushed to ECR
- A bigger next step would be #29333. This is also added to &9299
Fish scripts to sync images across repositories with scopeo
copy-helper-images.fish
#!/bin/fish
set VERSION $argv[1]
set FROM registry.gitlab.com/gitlab-org/gitlab-runner
set TO public.ecr.aws/gitlab
echo "syncing helper images for version $VERSION"
TAG=alpine-latest-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-x86_64-$VERSION-pwsh skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-x86_64-$VERSION-pwsh skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubi-fips-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-x86_64-$VERSION-pwsh skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore1809 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore2004 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore20H2 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore21H1 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
copy-runner-images.fish
#!/bin/fish
set VERSION $argv[1]
set FROM registry.gitlab.com/gitlab-org/gitlab-runner
set TO public.ecr.aws/gitlab
echo "syncing runner images for version $VERSION"
TAG=alpine-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.12-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.13-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.14-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.15-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=ubi-fips-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=ubuntu-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
usage:
$ fish copy-helper-images.fish v15.5.1
$ fish copy-runner-images.fish v15.5.1
Edited by Georgi N. Georgiev