Allow the CI_JOB_TOKEN access to the package API (at least on a project level) to DELETE and list (GET)
Created after a very useful discussion with @10io on #348690.
The Problem I need to be able to upload Terraform modules to the GitLab Terraform Module Registry within my CI tools. In simple cases this works, when I only need to upload a new version of the module, I can use the method explained here to do that, using the CI_JOB_TOKEN as a means of authentication.
However, I have some more complex scenarios where I need to upload the same version of a Terraform module multiple times. This happens, for example, when I would like to deploy the version of the terraform module from a feature branch and not have to create a new version number each time which would mean that I'd have to update my deployment scripts to point to that new version multiple times. So in other words, I'd like to have a "rolling" version of my modules in some cases.
This cannot be done by re-deploying the Terraform module as the registry will refuse overwriting an existing version. This is documented in the documentation. The documentation states that one cannot modify a package and suggest that the package must be deleted using the UI or the API and then re-uploaded (side note: I have created an issue requesting that re-uploading be considered using a special overwrite=true
parameter in the request: #348955).
However, the problem is that when calling the API using the CI_JOB_TOKEN to authenticate. It does not have access to any of the package registry endpoints. I simply get a 404 error stating that the project doesn't exist.
Here are some example curl commands that I have used both locally (using a PRIVATE-TOKEN to authenticate) as well as within the CI of the project owning the registry packages that I'm trying to delete (using JOB-TOKEN to authenticate):
# Gives a 404 when using JOB-TOKEN (in the CI for project 123) but works locally if using PRIVATE-TOKEN auth
- 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.mycompany.com/api/v4/projects/123/packages?package_type=terraform_module&per_page=100&package_name=gitlab-ci-test/local"'
# Gives a 404 when using JOB-TOKEN (in the CI for project 123) but works locally if using PRIVATE-TOKEN auth (if the package being deleted exists of course)
- 'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --request DELETE "https://gitlab.mycompany.com/api/v4/projects/123/packages/456"'
# Works in both CI and locally as far as the package in question doesn't already exist in the registry of course
- 'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ${TERRAFORM_MODULE_TGZ} ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_MODULE_SYSTEM}/${TERRAFORM_MODULE_VERSION}/file'
As @10io pointed out, there is a discrepancy in that the CI_JOB_TOKEN can download packages from the Package repository using the CLI tool endpoints but it cannot do the same using the GitLab Rest API.
But I would like to ask that this be taken even further and ask that the CI_JOB_TOKEN be given even more access to the GitLab Rest API. In my mind, one should be able to both GET (list) packages and DELETE them via the API using the CI_JOB_TOKEN.
@10io also pointed out that it should be taken into consideration whether it should gain "project access" to the packages or whether it should also have "group access" to the package registry via the API. I'll leave that discussion to others but in my use case, getting project level access would be sufficient.
Lastly, I'd like to mention that for my usage, I would not need any of this if #348955 where implemented. However, I do believe that allowing the CI job token access to the API, at least for project owned resources, is something that should be considered.