Remove container images from CI/CD
Problem to solve
The GitLab Container Registry allows users to build Docker images from GitLab CI. Typically, the pre-defined environment variables $CI_REGISTRY_USER
and $CI_REGISTRY_PASSWORD
are used for building and pushing containers to the registry from CI.
However, $CI_REGISTRY_USER
lacks permission to untag the images it has created. These images take up valuable storage space and clutter the user interface, making it difficult for teams to find and discover images/tags in their container registry.
Intended users
- Software Developer Often a developer will build an image for a specific branch/pipeline and never need it again.
- Systems AdministratorAllowing developers to untag their own images from CI will make the Systems Administrator's job of optimizing storage much easier, as they only have to worry about garbage collection, not identifying which images can be untagged.
Proposal
Allow $CI_REGISTRY_USER
to untag images that it has previously pushed from CI.
Example pipeline
stages:
- build
- release
- clean
variables:
# Use TLS https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled
DOCKER_HOST: tcp://docker:2376
DOCKER_TLS_CERTDIR: "/certs"
CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest
build branch image:
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- apk add curl
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- curl -L https://github.com/genuinetools/reg/releases/download/v0.13.0/reg-linux-amd64 > /bin/reg
script:
- curl -u "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" "https://gitlab.com/jwt/auth?account=$CI_REGISTRY_USER&scope=repository:$CI_REGISTRY_IMAGE:delete&service=container_registry"
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME"
except:
- master
release-image:
stage: release
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
- docker push $CONTAINER_RELEASE_IMAGE
only:
- master
delete branch image:
image: docker:latest
stage: clean
services:
- docker:dind
before_script:
- apk add curl
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- curl -L https://github.com/genuinetools/reg/releases/download/v0.13.0/reg-linux-amd64 > /bin/reg
- chmod +x /bin/reg
script:
- curl -u "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" "https://gitlab.com/jwt/auth?account=$CI_REGISTRY_USER&scope=repository:$CI_REGISTRY_IMAGE:delete&service=container_registry"
- /bin/reg -d -r $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD rm "$CI_PROJECT_PATH:$CI_COMMIT_REF_NAME"
except:
- master
Example video
Further details
Details from the original issue
This is a follow-up of #26465 (closed).
According to GitLab CI/CD Variables the two variables
$CI_REGISTRY_{USER|PASSWORD}
are to be used to log into a Docker registry with GitLab as its auth backend. The username to use to push containers to the GitLab Container Registrygitlab-ci-token
a.k.a.$CI_REGISTRY_USER
should be able to clean up pipeline artefacts it created in previous stages. As of GitLab 10.1.2 this fails (401, unauthorised) because the scope returned via the bearer token is[]
rather than[*]
.
Steps to reproduce
-
Create a pipeline with two stages in a project you own. One builds an image like this
docker build -t "$PROJECTURL/build" . docker push "$PROJECTURL/build"
-
... the other re-tags (in my case unit tests are running in-between these two stages)
docker pull "$PROJECTURL/build" docker tag "$PROJECTURL/build" "$PROJECTURL/release" docker push "$PROJECTURL/release" delete_image "$PROJECTURL/build"
-
Implement your
delete_image
-
Run the pipeline -> fails with 401
-
Override
$CI_REGISTRY_PASSWORD
with an access token (type "api") using a secret variable. -
Override
$CI_REGISTRY_USER
with your acronym using a secret variable. -
Run the pipeline -> succeeds
Permissions and Security
https://gitlab.com/gitlab-org/gitlab-ce/issues/58271 gave Developers the ability to untag a Docker image from the container registry. We will also allow $CI_REGISTRY_USER
to do the same.
Documentation
- Building Docker images from CI/CD: update example to include how to untag an image.
Testing
- Ensure that
$CI_REGISTRY_USER
can not delete images in bulk. - Ensure that
$CI_REGISTRY_USER
can only delete images it has created. (What happens in the case of multiple pipelines running at once?)
What does success look like, and how can we measure that?
Success looks like developers start to use GitLab CI to not only build images but to untag them and ensure that garbage collection cleans up as many images as possible. As the feature is adopted, we should see the same rate of image creation, but more untagged images.
Metrics
- number of images untagged / number of images created