Downloading artifacts returns 401 unauthorized

Problem

#master-broken is reporting a few incidents which shows job failed due to downloading artifact requests return 401 unauthorized.

Latest incidents:

• gitlab-org/quality/engineering-productivity/master-broken-incidents#1248 (closed)
• gitlab-org/quality/engineering-productivity/master-broken-incidents#1305 (closed)
• gitlab-org/quality/engineering-productivity/master-broken-incidents#1306 (closed)
• gitlab-org/quality/engineering-productivity/master-broken-incidents#1307 (closed)

For example this can be observed in this failing job: https://gitlab.com/gitlab-org/gitlab/-/jobs/4024261126

Error:

ERROR: Downloading artifacts from coordinator... unauthorized  host=gitlab.com id=4024260998 responseStatus=401 Unauthorized status=401 Unauthorized token=64_PZusJ
FATAL: permission denied       

The occurrences seem random as the first incident was resolved on a retry. However, I see this job failed on the retry attempt.

Proposal

Help needed to investigate what is causing these 401 responses, we will keep this issue updated if there is more data to help with troulbeshooting.