Artifact downloads fail with certain CI_JOB_TOKEN values

Status update 2022-07-26 (2:00 pm UTC)

The root cause is an update to the github.com/urfave/cli library to v1.22.2 which was done in this MR. We are working on reverting that change.

We will be releasing a patched version of the runner with this library reverted to undo this regression.

Summary

Sometimes the randomly generated CI/CD job token can begin with a leading dash (e.g. -AsdfABdagvf). If a job token happens to be created with this leading dash and that job token is then used for downloading previous job artifacts, the download will fail with the following:

ERROR: Downloading artifacts from coordinator... forbidden  id=JOB_ID_HERE responseStatus=403 Forbidden status=GET https://gitlab.com/api/v4/jobs/JOB_ID_HERE/artifacts: 403 Forbidden token=--id

It seems as if the - is being misinterpreted by the artifacts-downloader and causing the subsequent --id flag to be passed in as the --token value.
This has started cropping up with the 15.2 release of the runner specifically.

Steps to reproduce

This is very hard to reproduce, as it relies on the CI/CD randomly generated job token to have this leading dash. I have been unable to reproduce this on my own, but we have at least two tickets where the jobs encountering this error do have CI_JOB_TOKEN values beginning with the - character.

If you happen to get lucky and get a token with this value, the .gitlab-ci.yml below should replicate the issue.

.gitlab-ci.yml 
job:
  stage: test
  image: bash:latest
  script"
    - echo $CI_JOB_TOKEN > test.file
  artifacts:
    paths:
      - test.file

job2:
  stage: deploy
  image: bash:latest
  script:
    - echo ""
  needs:
    - job: "job"
      optional: false

Actual behavior

The --id flag is passed as the token to the artifacts-downloader if the CI/CD job token begins with a -.

Expected behavior

CI/CD job tokens beginning with the - character are properly are passed as the token properly.

Relevant logs and/or screenshots

Some interesting behavior is shown when calling the artifacts-downloader command directly within the helper image between 15.2 and 15.1.

Using 15.2's helper image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-7f093137

bash-5.1# /usr/bin/gitlab-runner-helper artifacts-downloader --url https://gitlab.redacted.com/ --token -Abajdbajdbajbd --id 123455
WARNING: Missing build ID (--id)                   
FATAL: Incomplete arguments

Using 15.1's helper image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-76984217

bash-5.1# /usr/bin/gitlab-runner-helper artifacts-downloader --url https://gitlab.redacted.com/ --token -Abajdbajdbajbd --id 123455
ERROR: Downloading artifacts from coordinator... forbidden  id=123455 responseStatus=403 Forbidden status=GET https://gitlab.redacted.com/api/v4/jobs/123455/artifacts?: 403 Forbidden token=-Abajdba
FATAL: permission denied

Environment description

Confirmed to be affecting Kubernetes based executors - untested on others at this time.

Used GitLab Runner version

Running with gitlab-runner 15.2.0 (7f093137)

Possible fixes

Add safeties for the - character to the artifacts-downloader utility.

Edited by Elliot Rushton