Efforts to once again support amazon ECR as a docker registry for GitLab Runner's images

Timeline:

A while back there was an idea to push only to our own registry. As per #29192 (comment 1181859546) I think this idea is scrapped.
We stopped pushing to ECR from our pipelines temporarily - https://gitlab.com/gitlab-sirt/incident_2382/-/issues/1
We opened https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29384 to track the effort of republishing to ECR.
Meanwhile we were pushing the images manually using scopeo. Find the very crude fish scripts I was using bellow.
We then started pushing to ECR again with !3568 (merged)
Shortly after we hit a limit on ECR repositories where images are capped to 10k - #29424 (closed)
The MR above was reverted and we once again stopped pushing to ECR
We requested from AWS to raise our limits from 10k to 20k as a temporary solution and also a good measure - https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29155#note_1171737236
While waiting for the limits to be raised we started working on #29070 in order to garbage collect any dev images that are no longer used since even with 20k limit our ECR repositories were bound to get filled up again.
I opened ggeorgiev_gitlab/runner-docker-images-registry-cleanup!1 (merged) so we could have an MVC which we can use to cleanup the images as soon as possible to unblock ECR pushing.
As of the time of writing creating this issue I just deleted 7k images from Amazon ECR based on the criteria listed here ggeorgiev_gitlab/runner-docker-images-registry-cleanup!1 (diffs)
All release images of GitLab Runner should be present in ECR as of 22.11.2022.
As a next step we will start running ggeorgiev_gitlab/runner-docker-images-registry-cleanup!1 (merged) on a schedule to keep the amount of images in ECR in a manageable level (issue coming up, this bullet point will also be updated)
The next immediate step will be to merge !3743 (closed) so images are once again automatically pushed to ECR
A bigger next step would be #29333. This is also added to &9299

Fish scripts to sync images across repositories with scopeo

copy-helper-images.fish

#!/bin/fish

set VERSION $argv[1]
set FROM registry.gitlab.com/gitlab-org/gitlab-runner
set TO public.ecr.aws/gitlab

echo "syncing helper images for version $VERSION"

TAG=alpine-latest-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine-latest-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.12-x86_64-$VERSION-pwsh skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.13-x86_64-$VERSION-pwsh skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.14-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=alpine3.15-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubi-fips-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-arm-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-arm64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-ppc64le-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-s390x-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=ubuntu-x86_64-$VERSION-pwsh skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore1809 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore2004 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore20H2 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG
TAG=x86_64-$VERSION-servercore21H1 skopeo copy --all docker://$FROM/gitlab-runner-helper:$TAG docker://$TO/gitlab-runner-helper:$TAG

copy-runner-images.fish

#!/bin/fish

set VERSION $argv[1]
set FROM registry.gitlab.com/gitlab-org/gitlab-runner
set TO public.ecr.aws/gitlab

echo "syncing runner images for version $VERSION"

TAG=alpine-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.12-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.13-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.14-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=alpine3.15-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=ubi-fips-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=ubuntu-$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG
TAG=$VERSION skopeo copy --all docker://$FROM:$TAG docker://$TO/gitlab-runner:$TAG

usage:

$ fish copy-helper-images.fish v15.5.1
$ fish copy-runner-images.fish v15.5.1

Edited Nov 23, 2022 by Georgi N. Georgiev | GitLab