Gitlab-runner instability

On 3rd January 2025 at around 00:05:10 CET we had a case where CI jobs were cancelled or failed for some reason:

• https://gitlab.com/gitlab-org/container-registry/-/jobs/8759108748:

Checking cache for 1.23-db66156752ece22d06419d9f3ece943e190c571d-1-protected...
Downloading cache from https://storage.googleapis.com/gitlab-com-runners-cache/project/13831684/1.23-db66156752ece22d06419d9f3ece943e190c571d-1-protected  ETag="91b66fbd3032734453297bcc39957677"
Successfully extracted cache
ERROR: Job failed: canceled

• https://gitlab.com/gitlab-org/container-registry/-/jobs/8759108353

registry/datastore/migrations/structure.sql: found 1 matching artifact files and directories 
WARNING: Upload request redirected                  location=https://gitlab.com/api/v4/jobs/8759108353/artifacts?artifact_format=zip&artifact_type=archive&expire_in=24h new-url=https://gitlab.com
WARNING: Retrying...                                context=artifacts-uploader error=request redirected
ERROR: Uploading artifacts as "archive" to coordinator... POST https://gitlab.com/api/v4/jobs/8759108353/artifacts: 403 Forbidden  id=8759108353 responseStatus=403 Forbidden status=403 token=glcbt-66
FATAL: permission denied                           
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

• job terminated via signal:
  • https://gitlab.com/gitlab-org/container-registry/-/jobs/8759109583/viewer#L6970
  • https://gitlab.com/gitlab-org/container-registry/-/jobs/8759109356/viewer#L9852

This is not a first time we get this kind of failures:

• #1440 (closed) - image pull timeout
• #1436 (closed) - minio host not found