The GitLab Container Registry allows users to build Docker images from GitLab CI. Typically, the pre-defined environment variables $CI_REGISTRY_USER and $CI_REGISTRY_PASSWORD are used for building and pushing containers to the registry from CI.
However, $CI_REGISTRY_USER lacks permission to untag the images it has created. These images take up valuable storage space and clutter the user interface, making it difficult for teams to find and discover images/tags in their container registry.
Intended users
Software Developer Often a developer will build an image for a specific branch/pipeline and never need it again.
Systems AdministratorAllowing developers to untag their own images from CI will make the Systems Administrator's job of optimizing storage much easier, as they only have to worry about garbage collection, not identifying which images can be untagged.
Proposal
Allow $CI_REGISTRY_USER to untag images that it has previously pushed from CI.
According to GitLab CI/CD Variables the two variables $CI_REGISTRY_{USER|PASSWORD} are to be used to log into a Docker registry with GitLab as its auth backend.
The username to use to push containers to the GitLab Container Registry
gitlab-ci-token a.k.a. $CI_REGISTRY_USER should be able to clean up pipeline artefacts it created in previous stages. As of GitLab 10.1.2 this fails (401, unauthorised) because the scope returned via the bearer token is [] rather than [*].
Steps to reproduce
Create a pipeline with two stages in a project you own. One builds an image like this
Ensure that $CI_REGISTRY_USER can not delete images in bulk.
Ensure that $CI_REGISTRY_USERcan only delete images it has created. (What happens in the case of multiple pipelines running at once?)
What does success look like, and how can we measure that?
Success looks like developers start to use GitLab CI to not only build images but to untag them and ensure that garbage collection cleans up as many images as possible. As the feature is adopted, we should see the same rate of image creation, but more untagged images.
Metrics
number of images untagged / number of images created
I don't think that the Docker API has some way to delete images from the registry via its CLI tool... But I would love a way to do this anyway, even if I need to do a CURL from the CI, I don't care...
Woha, so it really works! Great, thanks @lusitania, I was able to delete one image using Postman and a token retrieved using my personal login... So since #26465 (closed) is done and merged, I should be able to do it automatically in my CI!
I would like to add that the current scope of the token is repository:your/image:pull, so it's not usable to do anything apart pulling the images. I would like to use it to get info from the registry, like getting the SHA of a tag to delete it from the registry.
I have to add that I tried to use a Personal Access Token to obtain a JWT token from the registry with the * action allowed, and it was possible, BUT not from inside the CI, only from outside. It seems that there is some special fix in place to block a CI job from ever obtaining that privilege.
with #41084 (moved) I'm trying to get general more access from within a CI-job with a job-token, including the registry (push/pull/delete private/public projects), although it would then be an extended job token seen the security issues that would arise from adding more permissions to the current CI-JOB-TOKEN. (since CI-JOB_TOKEN is available in the environment also for unsafe executers like shell and ssh.)
Any support welcome!
I just ran into this issue as well, we want to build custom builder images during the first stage (with --cache-from so they don't need to do anything if the image definition hasn't changed), use these during later stages to build our project, then delete the temporary tag used for this pipeline. I was hoping that I could just use the reg tool in the cleanup job:
Would an acceptable solution be to provide a script in the environment that will hit our existing api to delete the tag? (see #21405 (moved)).
This would require the user and token for the api though, not the one provided to use with docker.
As stated before, I don't think docker cli supports this except using the dummy image trick describe in #21405 (moved).
This would facilitate the usage for the end user since all that would be needed is to call the script instead of building a custom solution and our api already supports this.
Not sure about limiting the tags to only those created in the build though. I think that's not feasible at the moment.
Would an acceptable solution be to provide a script in the environment that will hit our existing api to delete the tag? (see #21405 (moved)) This would require the user and token for the api though, not the one provided to use with docker.
No, that is the whole point of this issue. A script to automate this would be nice, but the main problem is that the ci user cannot untag its own images (or any image really)
Not sure about limiting the tags to only those created in the build though. I think that's not feasible at the moment.
I think a setting in the repository to specify if the ci user has pull/push rights (the default, as it is now) or complete control of the registry would solve this issue in an acceptable way
@lusitania Are you saying that if you hit DELETE /v2/<name>/manifests/<reference> on docker with $CI_REGISTRY_USER and $CI_REGISTRY_PASS it fails, but if you change the credentials it works?
"bulk deletion" cannot be prevented at this stage. The user deletes the tags directly on the registry api and not via gitlab api. The registry API only supports deleting a single image at a time (which deletes associated tags). I doesn't have "delete tag" API. That's what #21405 (moved) is trying to address.
In the scope of this issue, we cannot prevent the user from making a script that deleted all image/tags one by one.
The proposed MR !31796 (merged) is also not restricting it to only images the build has created as it would be impractical to achieve if not impossible at the moment.
@trizzi Sorry if I hijack this thread, but small question
Actually we just authorized CI_REGISTRY_USER to untag images
We tried to log in using CI_REGISTRY_USER and removing via docker image rm (as shown in the release post). Is this too the wrong approach? We get "Error: No such image ..." even though I can see an image with the exact same name in the container registry page.
@csvn No problem at all, I made a mistake in the image for the release post. :/ That command will actually only remove it from the local cache. We are correcting the blog post today and the documentation
Ahh, alright. Makes sense, I probably knew that somewhere in the back of my head since many docker commands only affect local files (e.g. push and pull are exceptions).
The snippet above looked "scarier" than if it was possible to use docker image rm, that's why I asked.
But get back an error message indicating that this operation is UNAUTHORIZED. Does this only work with the reg CLI tool. I've never used that before but the last release 0.16 seems to be almost one year old...
@trizzi I've tried a couple variations on the reg script and it doesn't seem to authorize properly. Neither the docker login method nor passing in username and password to reg (via -u and -p) seems to work. Are there hidden quirks to how the authentication works that I might be missing?
@tayloramurphy silly question, but do you have access to the project with developer or above privileges? Are you able to push images using CI_REGISTRY_USER/PASSWORD?
For context, we build some images on branches for an MR and then I was hoping to automatically delete them once the MR is merged using the environment on_stop trigger. The image tags / url looks correct in the response:
@tayloramurphy - First a little promotion for upcoming issues. This issue will hopefully make your use case a lot easier once we get it done. It will allow you to delete branch images once the merge request is accepted. gitlab#34073 (closed)
Second, this is moving beyond my skill level to troubleshoot. @ggelatti@axil Do you see what could be wrong with Taylor's script?
@glensc@tayloramurphy Just run the skopeo tool with verbose debug output and it seems that I've the same behaviour here. When comparing the non-working invocation in the CI with CI_REGISTRY_USER
Returning credentials from DockerAuthConfigUsing registries.d directory /etc/containers/registries.d for sigstore configuration Using \"default-docker\" configuration Using file:///var/lib/atomic/sigstoreLooking for TLS certificates and private keys in /etc/docker/certs.d/docker.ourgitlab.comGET https://docker.ourgitlab.com/v2/Ping https://docker.ourgitlab.com/v2/ status 401GET https://ourgitlab.com/jwt/auth?account=gitlab-ci-token&scope=repository%3Abuild-and-test%2Fasciidoctor%2Fci%2Fasciidoctor%3A%2A&service=container_registryIncreasing token expiration to: 60 secondsGET https://docker.ourgitlab.com/v2/build-and-test/asciidoctor/ci/asciidoctor/manifests/test-227308Failed to delete docker.ourgitlab.com/build-and-test/asciidoctor/ci/asciidoctor:test-227308: {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"build-and-test/asciidoctor/ci/asciidoctor\",\"Action\":\"pull\"}]}]}\n (401 Unauthorized)
with the output when executing it with a regular user and access token
Returning credentials from DockerAuthConfig Using registries.d directory /etc/containers/registries.d for sigstore configuration Using "default-docker" configuration Using file:///var/lib/atomic/sigstore Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.ourgitlab.com GET https://docker.ourgitlab.com/v2/ Ping https://docker.ourgitlab.com/v2/ status 401 GET https://ourgitlab.com/jwt/auth?account=kofleri&scope=repository%3Abuild-and-test%2Fasciidoctor%2Fci%2Fasciidoctor%3A%2A&service=container_registry Increasing token expiration to: 60 seconds GET https://docker.ourgitlab.com/v2/build-and-test/asciidoctor/ci/asciidoctor/manifests/test-227308 DELETE https://docker.ourgitlab.com/v2/build-and-test/asciidoctor/ci/asciidoctor/manifests/sha256:888f9ee63443377ca811668711b48a35385ab08a8af82faeddab2866236d7943 Deleting /var/lib/atomic/sigstore/build-and-test/asciidoctor/ci/asciidoctor@sha256=888f9ee63443377ca811668711b48a35385ab08a8af82faeddab2866236d7943/signature-1
One can see that it does not fail at the DELETE http operation but already one operation before, when fetching the manifest. Any ideas?
GET https://ourgitlab.com/jwt/auth?account=gitlab-ci-token&scope=repository%3Abuild-and-test%2Fasciidoctor%2Fci%2Fasciidoctor%3A%2A&service=container_registry
I don't see a proper scope here. It should be something like repository:path/to/repo:delete
@tayloramurphy I had the same issue trying to get this working today. You're trying to remove $CI_REGISTRY_IMAGE/airflow_image:$CI_COMMIT_REF_NAME, but $CI_REGISTRY_IMAGE includes the registry as well (i.e. in your case it looks like registry.gitlab.com/gitlab-data/data-image based on the error message). What worked for me was using $CI_PROJECT_PATH instead (which would be gitlab-data/data-image), and still specifying the registry separately, i.e.
Everyone, we're looking at a universal solution around job token permissions to unlock this along with several other issues in one go, and in a consistent way. The idea is that there will be a project setting that allows you to finely control CI_JOB_TOKEN permissions. There is a brief thread at gitlab#20416 (comment 216069060) that it would be great to have your feedback in, especially if you feel that does not meet your use case.
@glensc The scope there should be delete not a pull I believe. Even though I think a pull should be successfully authenticated as well, that's not what's expected in this context I believe.
I added this for debug only so I can see that the authentication response is correct.
The way it works is the registry api returns a redirect saying "hey, authenticate with gitlab first", then you go there and ask for a token. You then pass this token back to the registry.
Can the implemented feature delete tags, not images? I have dummy image (one image, but tagged twice) and example with reg util deletes both tags.
My case: I build the image and tag it with commit SHA and branch name: commit SHA tag uses in CI flow, and branch tag uses in CD flow. When all checks in CI passed I delete SHA tag and it keeps my registry clean.
@b0g3r You can delete tags. Here is an example CI pipeline that demonstrates building and tagging an image with CI_COMMIT_REF but you can use COMMIT_SHA instead. Hopefully you can use this an example. Also here is a video of this pipeline running in real-time.
There is an issue somewhere about this behaviour from the UI as well. I believe it is fundamentally a limitation of the API provided to delete images not allowing just untagging one tag pointing to the image. A workaround I have been using is to have separate /branch and /commit sub-docker-repositories to which the different kinds of tags are pushed, deleting an image from one repository doesn’t affect other repositories.
What would be an alternative be? I assume you don't mean unauthenticated registry changes. reg also relies on credentials (as does docker login), but we can't use the predefined $CI_REGISTRY_USER/$CI_REGISTRY_PASSWORD for deletes until the current permission issue is fixed. In the meantime we can just use our personal credentials or create new API tokens.
I waited for native docker registry deleting
Does such a thing exist? I thought even in Docker Hub, deletes have to be done via the UI or HTTP request.
I asked this above, but what's the point of using reg at all? Is it just the fact that you can reference image names directly instead of finding the registry ID needed to use the REST API?
If so, you can easily get those:
Log into GitLab in your browser
View the list of projects you're a member of at https://gitlab.com/api/v4/projects/?membership=true
Note the id of the desired project (this is the value CI_PROJECT_ID is set to during CI jobs)
Get the list of images ("registry repositories") for the desired project at https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/registry/repositories
Get the id of the desired image and set it up as a custom environment variable (e.g. MY_REGISTRY_REPOSITORY_ID)
Set up another custom environment variable (e.g. MY_REGISTRY_TOKEN) as a token with api privileges or your account password (CI_REGISTRY_PASSWORD will have insufficient privileges for image/tag deletion until the present issue is fixed)
Now assuming the name of the tag you want to delete is stored in MY_TAG_NAME (in your case you'd use CI_COMMIT_REF_NAME instead) make a tag delete request: curl --request DELETE --header "PRIVATE-TOKEN: $MY_REGISTRY_TOKEN" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/registry/repositories/$MY_REGISTRY_REPOSITORY_ID/tags/$MY_TAG_NAME
P.S. docker login is needed before a docker push, but is not needed for deletion
@frannelk, yes. Below is the custom stage+job we're currently using to delete temporary tags of our images; most of it should be general enough to reuse other than our image names of cp, cp_client_builder, and elasticsearch.
# Clean up temporary images pushed in `docker_build` jobdocker_registry_cleanup:stage:cleanup# Run this job even if there were failures in preceding stageswhen:always<<:*except__release_branchimage:alpine:3# Delete the commit-specific images built in the `docker_build` job via the REST API (see https://docs.gitlab.com/ee/api/container_registry.html)script:# TODO: Generate and use JWTs instead of `DOCKER_REGISTRY_PASSWORD` (see https://gitlab.com/gitlab-org/gitlab-foss/issues/26465s)-apk add --no-cache curl jq# Get the repository IDs for the images to be cleaned-REGISTRY_REPOSITORIES_ENDPOINT="$CI_API_V4_URL/projects/$CI_PROJECT_ID/registry/repositories"-DOCKER_REGISTRY_REPOSITORIES_JSON=$(curl --silent --header PRIVATE-TOKEN:$DOCKER_REGISTRY_PASSWORD $REGISTRY_REPOSITORIES_ENDPOINT)-CP_REGISTRY_REPOSITORY_ID=$(echo $DOCKER_REGISTRY_REPOSITORIES_JSON | jq '.[] | select(.name == "cp").id')-CP_CLIENT_BUILDER_REGISTRY_REPOSITORY_ID=$(echo $DOCKER_REGISTRY_REPOSITORIES_JSON | jq '.[] | select(.name == "cp_client_builder").id')-ELASTICSEARCH_REGISTRY_REPOSITORY_ID=$(echo $DOCKER_REGISTRY_REPOSITORIES_JSON | jq '.[] | select(.name == "elasticsearch").id')# Delete the `TEMPORARY_BUILD_TAG` for the images-curl --silent --output /dev/null --write-out %{http_code} --request DELETE --header PRIVATE-TOKEN:$DOCKER_REGISTRY_PASSWORD $REGISTRY_REPOSITORIES_ENDPOINT/$CP_REGISTRY_REPOSITORY_ID/tags/$TEMPORARY_BUILD_TAG | grep -e '^200$'-curl --silent --output /dev/null --write-out %{http_code} --request DELETE --header PRIVATE-TOKEN:$DOCKER_REGISTRY_PASSWORD $REGISTRY_REPOSITORIES_ENDPOINT/$CP_CLIENT_BUILDER_REGISTRY_REPOSITORY_ID/tags/$TEMPORARY_BUILD_TAG | grep -e '^200$'-curl --silent --output /dev/null --write-out %{http_code} --request DELETE --header PRIVATE-TOKEN:$DOCKER_REGISTRY_PASSWORD $REGISTRY_REPOSITORIES_ENDPOINT/$ELASTICSEARCH_REGISTRY_REPOSITORY_ID/tags/$TEMPORARY_BUILD_TAG | grep -e '^200$'
We should not allow $CI_REGISTRY_USER to delete tags in bulk, as this is a highly destructive operation and one could effectively remove all images by mistake.
But I'm not sure that's desirable? In lieu of having an ability to configure an image retention policy, allowing a CI scheduled job to perform bulk tag deletion would allow building custom image retention in a reasonable way.
While this solution looks really great, until Gitlab implements something cleaner, the unfortunate thing here is that the Docker image has only a latest tag, which makes me not really confident to use that image.
As you seem to be the image owner, maybe you could fix that?
Can anyone share how does their working JWT look? I tried this snippet from the documentation (using latest instead of CI_COMMIT_SHORT_SHA), but I got back 401: