2025-03-06: Increased 403 rate for package registry

Customer Impact

We have received several tickets from customers who are seeing jobs failing with a 403 when trying to download packages using the CI_JOB_TOKEN.

https://gitlab.zendesk.com/agent/tickets/610198

https://gitlab.zendesk.com/agent/tickets/610228

https://gitlab.zendesk.com/agent/tickets/610223

One customer worked around the problem by replacing the CI_JOB_TOKEN with a personal access token.

https://gitlab.zendesk.com/agent/tickets/610153

In the logs we can see a slight increase in 403 errors for the package registry.

Screenshot_2025-03-06_at_17.06.45

source

Setting this as a severity 3 as there looks like there is a potential workaround.

Current Status

We've narrowed down the issue to CI_JOB_TOKEN failures for a small subset of users for Python Packages.

A MR with a fix has been put in place and is now rolled out to production.

📚 References and helpful links

Recent Events (available internally only):

Deployment Guidance

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.

Edited by Nick Duff