Publish docker images to ECR public
What does this MR do?
Publish docker images to ECR public
Why was this MR needed?
We GitLab Runner docker images to be published inside of https://gallery.ecr.aws/gitlab/gitlab-runner automatically.
What's the best way to test this MR?
Linux
- Have an aws account set up.
- If you are using the development aws account, a registry is already created
public.ecr.aws/j9a4b5w2. If you are using your own account create a new repository following the official documentation - Assuming that you have aws cli installed and configured. Get a temporary token:
export ECR_PUBLIC_PASSWORD=$(docker run --rm -it -v ~/.aws:/root/.aws -v $PWD:/aws amazon/aws-cli ecr-public get-login-password --region us-east-1) - Build helper images:
make helper-dockerarchive - Build runner binaries:
make runner-bin - Build and release images:
ECR_PUBLIC_REGISTRY=public.ecr.aws/j9a4b5w2 PUBLISH_IMAGES=true PUSH_TO_ECR_PUBLIC=true TARGET_ARCHS="amd64" ./ci/release_docker_images
script log
amd64:
Building and pushing image: dockerfiles/runner/ubuntu
docker-buildx
Successfully created context "docker-buildx"
buildx-builder
WARNING! Your password will be stored unencrypted in /Users/steve/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[+] Building 262.4s (12/12) FINISHED
=> [internal] booting buildkit 28.6s
=> => pulling image moby/buildkit:buildx-stable-1 26.8s
=> => creating container buildx_buildkit_buildx-builder0 1.8s
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 899B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:20.04 2.6s
=> [1/6] FROM docker.io/library/ubuntu:20.04@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c 8.5s
=> => resolve docker.io/library/ubuntu:20.04@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c 0.0s
=> => sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601 162B / 162B 0.2s
=> => sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d 847B / 847B 0.4s
=> => sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8 28.56MB / 28.56MB 2.3s
=> => extracting sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8 6.0s
=> => extracting sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d 0.1s
=> => extracting sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601 0.0s
=> [internal] load build context 8.0s
=> => transferring context: 146.20MB 8.0s
=> [2/6] RUN apt-get update -y && apt-get install -y --no-install-recommends apt-transport-https ca-certificates curl git wget tzdata openssh-client && rm -rf /var/lib/apt/lists/* 19.5s
=> [3/6] COPY gitlab-runner_*.deb checksums-* install-deps install-gitlab-runner /tmp/ 0.6s
=> [4/6] RUN /tmp/install-deps "linux/amd64" "0.16.2" "1.2.2" "2.11.0" 29.7s
=> [5/6] COPY entrypoint / 0.0s
=> [6/6] RUN chmod +x /entrypoint 0.2s
=> exporting to image 171.9s
=> => exporting layers 15.5s
=> => exporting manifest sha256:6ded72b2f59b4a5b278d3c1967a98751188f92657a7d18948c6bc021f5a2d5ae 0.0s
=> => exporting config sha256:a0e5777e079cc07f4445ca06515a8f0116f35db0d1771be24b6a725313c5c508 0.0s
=> => pushing layers 0.7s
=> => pushing manifest for public.ecr.aws/j9a4b5w2/gitlab-runner:ubuntu-bleeding 0.7s
=> => pushing manifest for public.ecr.aws/j9a4b5w2/gitlab-runner:bleeding 0.6s
Removing login credentials for public.ecr.aws
Building and pushing image: dockerfiles/runner/alpine
docker-buildx
Successfully created context "docker-buildx"
buildx-builder
WARNING! Your password will be stored unencrypted in /Users/steve/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[+] Building 113.6s (14/14) FINISHED
=> [internal] booting buildkit 12.7s
=> => pulling image moby/buildkit:buildx-stable-1 11.7s
=> => creating container buildx_buildkit_buildx-builder0 1.0s
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 937B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:3.12.0 2.5s
=> [1/8] FROM docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 1.3s
=> => resolve docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s
=> => sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.80MB / 2.80MB 0.6s
=> => extracting sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 0.7s
=> [internal] load build context 6.6s
=> => transferring context: 107.48MB 6.5s
=> [2/8] RUN adduser -D -S -h /home/gitlab-runner gitlab-runner 0.6s
=> [3/8] RUN apk add --no-cache bash ca-certificates git tzdata openssh-client 5.6s
=> [4/8] COPY gitlab-runner-linux-* /usr/bin/ 0.4s
=> [5/8] COPY checksums-* install-deps install-gitlab-runner /tmp/ 0.1s
=> [6/8] RUN apk add --no-cache --virtual .fetch-deps wget && /tmp/install-deps "linux/amd64" "0.16.2" "1.2.2" "2.11.0" && apk del .fetch-deps 10.3s
=> [7/8] COPY entrypoint / 0.0s
=> [8/8] RUN chmod +x /entrypoint 0.1s
=> exporting to image 79.3s
=> => exporting layers 14.6s
=> => exporting manifest sha256:753b28bd6c9b386020f925a0d81d3b90b723c4cd956b4beaf83d884629e0b04e 0.0s
=> => exporting config sha256:25bc27468d6e2ee45179a546e8f64481563edac16b7a087a05cd7bd966f0627a 0.0s
=> => pushing layers 64.0s
=> => pushing manifest for public.ecr.aws/j9a4b5w2/gitlab-runner:alpine-bleeding 0.6s
Removing login credentials for public.ecr.aws
WARNING! Your password will be stored unencrypted in /Users/steve/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Importing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:x86_64-c7dfd900
sha256:c062e8663ca97954474ea167517b14c771a7a9e6c8903e40ef40beed15def4a7
Importing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm-c7dfd900
sha256:57b7b37ad2a0221dbfaadfa363881e36d9a44cfec31f42f2b6dee6dc0ffb08ee
Importing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm64-c7dfd900
sha256:3c6f6a51269c8fc110343613c6d386bde0b274ff5162c01ddc43d2839ba7f523
Importing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:s390x-c7dfd900
sha256:3a0066c6ace843397a68e340db6b61aa38a7f921542b157744d2b4a1d1442694
Tagging image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:x86_64-bleeding
Tagging image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm-bleeding
Tagging image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm64-bleeding
Tagging image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:s390x-bleeding
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:x86_64-c7dfd900
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
11ebb9b84bc0: Pushed
x86_64-c7dfd900: digest: sha256:ba3e5ac5cb369ab4c1d5db9d52fcf539673fa09cdf4661568fd5c258ef44d9c9 size: 528
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm-c7dfd900
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
5d310e0ed4d0: Pushed
arm-c7dfd900: digest: sha256:33547b364a3b9c1bd5551607a1c64f3e39a0d4e6b554e4731edce20516b70538 size: 528
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm64-c7dfd900
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
851eefc7c7ea: Pushed
arm64-c7dfd900: digest: sha256:c8dd7f3d9a10a2f80972c069c2c04c179ebeebda592318d1591f73886069675a size: 528
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:s390x-c7dfd900
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
6baf0f2e220e: Pushed
s390x-c7dfd900: digest: sha256:b670dd27d0c47e80910eb1c1955c72cea2f676575d88b688120311c5607ac211 size: 528
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:x86_64-bleeding
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
11ebb9b84bc0: Layer already exists
x86_64-bleeding: digest: sha256:ba3e5ac5cb369ab4c1d5db9d52fcf539673fa09cdf4661568fd5c258ef44d9c9 size: 528
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm-bleeding
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
5d310e0ed4d0: Layer already exists
arm-bleeding: digest: sha256:33547b364a3b9c1bd5551607a1c64f3e39a0d4e6b554e4731edce20516b70538 size: 528
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:arm64-bleeding
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
851eefc7c7ea: Layer already exists
arm64-bleeding: digest: sha256:c8dd7f3d9a10a2f80972c069c2c04c179ebeebda592318d1591f73886069675a size: 528
Pushing image: public.ecr.aws/j9a4b5w2/gitlab-runner-helper:s390x-bleeding
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
6baf0f2e220e: Layer already exists
s390x-bleeding: digest: sha256:b670dd27d0c47e80910eb1c1955c72cea2f676575d88b688120311c5607ac211 size: 528
Removing login credentials for public.ecr.awsimage published in ECR
Windows
-
Start the windows environment:
vagrant up -
RDP inside of the machine:
vagrant rdp -
Set up the following variables
powershell
Set-Item -Path env:PUBLISH_IMAGES -Value true Set-Item -Path env:PUSH_TO_ECR_PUBLIC -Value true Set-Item -Path env:ECR_PUBLIC_PASSWORD -Value xxxx # Use the same value from the Linux token Set-Item -Path env:ECR_PUBLIC_REGISTRY -Value public.ecr.aws/j9a4b5w2 Set-Item -Path env:CI_REGISTRY_IMAGE -Value registry.gitlab.com/gitlab-org/gitlab-runner Set-Item -Path env:CI_REGISTRY -Value registry.gitlab.com Set-Item -Path env:CI_REGISTRY_IMAGE -Value registry.gitlab.com/steveazz/playground Set-Item -Path env:CI_REGISTRY_PASSWORD -Value xxx Set-Item -Path env:CI_REGISTRY_USER -Value xxx Set-Item -Path env:PWSH_VERSION -Value 7.0.3 Set-Item -Path env:PWSH_256_CHECKSUM -Value ad3b4a868d1b7e47a1048e1eb20f7f782d9b95d5066d79a25d02ccc4dd14e79f Set-Item -Path env:GIT_VERSION_BUILD -Value 1 Set-Item -Path env:GIT_VERSION -Value 2.27.0 Set-Item -Path env:GIT_256_CHECKSUM -Value bdb8c1560eca39f4d99b07705006c00d94a3f8612501046a0f89353afc5307fa Set-Item -Path env:GIT_LFS_256_CHECKSUM -Value ca05b0b9bd39d99665045776a5d0d5c7bfa2605a6118c2f489bfd40c2c3e4f2c Set-Item -Path env:GIT_LFS_VERSION -Value 2.11.0 Set-Item -Path env:SKIP_CLEANUP -Value true Set-Item -Path env:WINDOWS_VERSION -Value servercore1809 -
Build the image:
.\ci\build_release_windows_images.ps1
script log
PS C:\GitLab-Runner> .\ci\build_release_windows_images.ps1
Build image for x86_64_servercore1809
Directory: C:\GitLab-Runner\dockerfiles\runner-helper
Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 12/7/2020 5:11 AM binaries
Sending build context to Docker daemon 42.99MB
Step 1/25 : ARG BASE_IMAGE_TAG
Step 2/25 : FROM ${BASE_IMAGE_TAG} as builder
---> 2352228ff6bc
Step 3/25 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
---> Running in 775fdf7693bb
Removing intermediate container 775fdf7693bb
---> 911954ca7741
Step 4/25 : ARG GIT_VERSION
---> Running in ac324fb71f88
Removing intermediate container ac324fb71f88
---> 7175ffeb3dce
Step 5/25 : ARG GIT_VERSION_BUILD
---> Running in 581a003e31be
Removing intermediate container 581a003e31be
---> 77fbf86b8e0a
Step 6/25 : ARG GIT_256_CHECKSUM
---> Running in 7d344b9d3fb2
Removing intermediate container 7d344b9d3fb2
---> 23c8e8d6be90
Step 7/25 : ARG GIT_LFS_VERSION
---> Running in ea14cd6bea36
Removing intermediate container ea14cd6bea36
---> 7c29ba30a28c
Step 8/25 : ARG GIT_LFS_256_CHECKSUM
---> Running in b47e1642db57
Removing intermediate container b47e1642db57
---> 7f29e83e5241
Step 9/25 : ARG PWSH_VERSION
---> Running in cee16783349a
Removing intermediate container cee16783349a
---> 82a1f4a68b43
Step 10/25 : ARG PWSH_256_CHECKSUM
---> Running in eb2b00a66f07
Removing intermediate container eb2b00a66f07
---> 8b6ff1f41754
Step 11/25 : RUN New-Item -ItemType directory -Path C:\Downloads; [Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12; Invoke-Webrequest "https://github.com/git-for-windows/git/releases/download/v${Env:GIT_VERSION}.windows.${Env:GIT_VERSION_BUILD}/MinGit-${Env:GIT_VERSION}-64-bit.zip" -OutFile git.zip -UseBasicParsing; Invoke-Webrequest "https://github.com/git-lfs/git-lfs/releases/download/v${Env:GIT_LFS_VERSION}/git-lfs-windows-amd64-v${Env:GIT_LFS_VERSION}.zip" -OutFile git-lfs.zip -UseBasicParsing; Invoke-Webrequest "https://github.com/PowerShell/PowerShell/releases/download/v${Env:PWSH_VERSION}/PowerShell-${Env:PWSH_VERSION}-win-x64.msi" -OutFile C:\Downloads\pwsh.msi -UseBasicParsing
---> Running in 1ab9c40a2fbd
Directory: C:\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 12/7/2020 6:03 AM Downloads
Removing intermediate container 1ab9c40a2fbd
---> a91aa2fbb941
Step 12/25 : COPY [".\\helpers\\checksum.ps1", ".\\"]
---> f6a67176ba62
Step 13/25 : RUN powershell -File .\checksum.ps1 -TargetFile git.zip -ExpectedHash ${Env:GIT_256_CHECKSUM}; powershell -File .\checksum.ps1 -TargetFile git-lfs.zip -ExpectedHash ${Env:GIT_LFS_256_CHECKSUM}; powershell -File .\checksum.ps1 -TargetFile C:\Downloads\pwsh.msi -ExpectedHash ${Env:PWSH_256_CHECKSUM}
---> Running in 3dddd5b6f679
SHA256 checksum for git.zip is valid
SHA256 checksum for git-lfs.zip is valid
SHA256 checksum for C:\Downloads\pwsh.msi is valid
Removing intermediate container 3dddd5b6f679
---> 94bcc6cae216
Step 14/25 : RUN New-Item -ItemType directory -Path C:\Temp; Expand-Archive -Path git.zip -DestinationPath C:\Temp\git; Expand-Archive -Path git-lfs.zip -DestinationPath C:\Temp\git-lfs
---> Running in d928e95783af
Directory: C:\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 12/7/2020 6:04 AM Temp
Removing intermediate container d928e95783af
---> bd559d5689f6
Step 15/25 : COPY [".\\binaries\\gitlab-runner-helper.x86_64-windows.exe", "C:\\Temp\\gitlab-runner-helper\\gitlab-runner-helper.exe"]
---> d01055cfe670
Step 16/25 : FROM ${BASE_IMAGE_TAG}
---> 2352228ff6bc
Step 17/25 : USER ContainerAdministrator
---> Running in e6288c77026d
Removing intermediate container e6288c77026d
---> e27474fa90d8
Step 18/25 : COPY --from=builder ["C:\\Temp", "C:\\Program Files\\"]
---> af6c3444e184
Step 19/25 : COPY --from=builder ["C:\\Downloads", "C:\\Downloads\\"]
---> a553b5956bc2
Step 20/25 : RUN msiexec.exe /package "C:\Downloads\pwsh.msi" /quiet REGISTER_MANIFEST=1 && rmdir /s /q "C:\Downloads"
---> Running in 250cc5d9b1ab
Removing intermediate container 250cc5d9b1ab
---> 6ad32204280d
Step 21/25 : RUN pwsh --version
---> Running in 101cfdccd35d
PowerShell 7.0.3
Removing intermediate container 101cfdccd35d
---> 8d862a45e9d5
Step 22/25 : SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
---> Running in 2f9043e7ec27
Removing intermediate container 2f9043e7ec27
---> afb3c9d2b7dd
Step 23/25 : RUN [Environment]::SetEnvironmentVariable('Path', $env:Path + ';C:\Program Files\git\cmd;C:\Program Files\git-lfs;C:\Program Files\gitlab-runner-helper', [EnvironmentVariableTarget]::Machine); $env:Path = [System.Environment]::GetEnvironmentVariable('Path', 'Machine'); $env:GIT_CONFIG_NOSYSTEM=1; & $env:ProgramFiles\git\cmd\git.exe config --system --unset-all include.path; & $env:ProgramFiles\git\cmd\git.exe config --system core.longpaths true; & $env:ProgramFiles\git\cmd\git.exe lfs install --skip-repo
---> Running in 124cc7f07fdc
Git LFS initialized.
Removing intermediate container 124cc7f07fdc
---> 08ee66a904c5
Step 24/25 : COPY [".\\helpers\\entrypoint.cmd", ".\\"]
---> f16f64c7abd3
Step 25/25 : ENTRYPOINT ["entrypoint.cmd"]
---> Running in 7f0e344b87f4
Removing intermediate container 7f0e344b87f4
---> 39e5ffa0040b
Successfully built 39e5ffa0040b
Successfully tagged gitlab/gitlab-runner-helper:x86_64-c7dfd900-servercore1809
Successfully tagged registry.gitlab.com/steveazz/playground/gitlab-runner-helper:x86_64-c7dfd900-servercore1809
Successfully tagged public.ecr.aws/j9a4b5w2/gitlab-runner-helper:x86_64-c7dfd900-servercore1809
Login registry registry.gitlab.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in C:\Users\Administrator\.docker\config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Push x86_64-c7dfd900-servercore1809
The push refers to repository [registry.gitlab.com/steveazz/playground/gitlab-runner-helper]
69c80a9b8b44: Pushed
6c499924cbb4: Pushed
657a0cb0d43a: Pushed
e591fab9bc67: Pushed
abb351693055: Pushed
ca6df2014535: Pushed
a4f1d0da78fa: Pushed
6944d12b2855: Pushed
b392b9426635: Pushed
1199ba601e60: Skipped foreign layer
a7ba3db29ebb: Skipped foreign layer
x86_64-c7dfd900-servercore1809: digest: sha256:0f5c24a5a58ef5fd7b8f0fb2b4879524a122a7c15f9815730f1a1d695fc490ec size: 2991
Logout registry registry.gitlab.com
Removing login credentials for registry.gitlab.com
Login registry public.ecr.aws/j9a4b5w2
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in C:\Users\Administrator\.docker\config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Push x86_64-c7dfd900-servercore1809
The push refers to repository [public.ecr.aws/j9a4b5w2/gitlab-runner-helper]
69c80a9b8b44: Pushed
6c499924cbb4: Pushed
657a0cb0d43a: Pushed
e591fab9bc67: Pushed
abb351693055: Pushed
ca6df2014535: Pushed
a4f1d0da78fa: Pushed
6944d12b2855: Pushed
b392b9426635: Pushed
1199ba601e60: Skipped foreign layer
a7ba3db29ebb: Skipped foreign layer
x86_64-c7dfd900-servercore1809: digest: sha256:0f5c24a5a58ef5fd7b8f0fb2b4879524a122a7c15f9815730f1a1d695fc490ec size: 2991
Logout registry public.ecr.aws/j9a4b5w2
Removing login credentials for public.ecr.awsecr image pushed
What are the relevant issue numbers?
Reference #27269 (closed)
Edited by Steve Xuereb