Skip to content

Remove container images from CI/CD

Problem to solve

The GitLab Container Registry allows users to build Docker images from GitLab CI. Typically, the pre-defined environment variables $CI_REGISTRY_USER and $CI_REGISTRY_PASSWORD are used for building and pushing containers to the registry from CI.

However, $CI_REGISTRY_USER lacks permission to untag the images it has created. These images take up valuable storage space and clutter the user interface, making it difficult for teams to find and discover images/tags in their container registry.

Intended users

  • Software Developer Often a developer will build an image for a specific branch/pipeline and never need it again.
  • Systems AdministratorAllowing developers to untag their own images from CI will make the Systems Administrator's job of optimizing storage much easier, as they only have to worry about garbage collection, not identifying which images can be untagged.

Proposal

Allow $CI_REGISTRY_USER to untag images that it has previously pushed from CI.

Example pipeline

stages:
  - build
  - release
  - clean

variables:
  # Use TLS https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"
  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest

build branch image:
  image: docker:latest
  stage: build
  services:
    - docker:dind
  before_script:
    - apk add curl
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - curl -L https://github.com/genuinetools/reg/releases/download/v0.13.0/reg-linux-amd64 > /bin/reg
  script:
    - curl -u "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" "https://gitlab.com/jwt/auth?account=$CI_REGISTRY_USER&scope=repository:$CI_REGISTRY_IMAGE:delete&service=container_registry"
    - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME"
  except:
    - master

release-image:
  stage: release
  script:
    - docker pull $CONTAINER_TEST_IMAGE
    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
    - docker push $CONTAINER_RELEASE_IMAGE
  only:
    - master

delete branch image:
  image: docker:latest
  stage: clean
  services:
    - docker:dind
  before_script:
    - apk add curl
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - curl -L https://github.com/genuinetools/reg/releases/download/v0.13.0/reg-linux-amd64 > /bin/reg
    - chmod +x /bin/reg
  script:
    - curl -u "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" "https://gitlab.com/jwt/auth?account=$CI_REGISTRY_USER&scope=repository:$CI_REGISTRY_IMAGE:delete&service=container_registry"
    - /bin/reg -d -r $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD rm "$CI_PROJECT_PATH:$CI_COMMIT_REF_NAME"
  except:
    - master

Example video

Further details

Details from the original issue

This is a follow-up of #26465 (closed).

According to GitLab CI/CD Variables the two variables $CI_REGISTRY_{USER|PASSWORD} are to be used to log into a Docker registry with GitLab as its auth backend. The username to use to push containers to the GitLab Container Registry gitlab-ci-token a.k.a. $CI_REGISTRY_USER should be able to clean up pipeline artefacts it created in previous stages. As of GitLab 10.1.2 this fails (401, unauthorised) because the scope returned via the bearer token is [] rather than [*].

Steps to reproduce

  1. Create a pipeline with two stages in a project you own. One builds an image like this

    docker build -t "$PROJECTURL/build" .
docker push "$PROJECTURL/build"

  2. ... the other re-tags (in my case unit tests are running in-between these two stages)

    docker pull "$PROJECTURL/build"
docker tag "$PROJECTURL/build" "$PROJECTURL/release"
docker push "$PROJECTURL/release"
delete_image "$PROJECTURL/build"

  3. Implement your delete_image

  4. Run the pipeline -> fails with 401

  5. Override $CI_REGISTRY_PASSWORD with an access token (type "api") using a secret variable.

  6. Override $CI_REGISTRY_USER with your acronym using a secret variable.

  7. Run the pipeline -> succeeds

Permissions and Security

https://gitlab.com/gitlab-org/gitlab-ce/issues/58271 gave Developers the ability to untag a Docker image from the container registry. We will also allow $CI_REGISTRY_USER to do the same.

Documentation

Testing

  • Ensure that $CI_REGISTRY_USER can not delete images in bulk.
  • Ensure that $CI_REGISTRY_USERcan only delete images it has created. (What happens in the case of multiple pipelines running at once?)

What does success look like, and how can we measure that?

Success looks like developers start to use GitLab CI to not only build images but to untag them and ensure that garbage collection cleans up as many images as possible. As the feature is adopted, we should see the same rate of image creation, but more untagged images.

Metrics

  • number of images untagged / number of images created

Links / references

Edited by Tim Rizzi