Download of upstream project artifact with CI_JOB_TOKEN fails or get's artifact from previous pipeline
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=25071)
</details>
<!--IssueSummary end-->
### Summary
I have a pipeline triggering a downstream project using CI_JOB_TOKEN and waiting for the downstream job to complete. The downstream pipeline uses the api to get an artifact from a job that has already succeded in the upstream pipeline. I'm finding that I'll get a 401 if the upstream has never passed or it will get the artifact from the previous successful run if available.
I would expect the CI_JOB_TOKEN to only be able to access within the pipeline run of a parent. I would also expect it to get the artifact of a job that has passed even if the entire pipeline is complete.
I've had to work around this by querying the upstream project to get the job id of the job with the archive.
### Steps to reproduce
Example from the docs fails if the upstream project has not successfully finished any runs.
https://docs.gitlab.com/ee/api/jobs.html#download-the-artifacts-archive
```
curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test"
```
Workaround:
The upstream project includes project_id, job_name and pipeline_id as trigger vars.
```
#!/usr/bin/env bash
set -x
set -e
GITLAB_URL=https://gitlab.my-company.example
UPSTREAM_PROJECT_ID=5103
#FAILS: curl -v -k --header "JOB-TOKEN: ${CI_JOB_TOKEN}" -o artifacts_test.zip "${GITLAB_URL}/api/v4/projects/5103/jobs/artifacts/master/download?job=build"
echo ${UPSTREAM_PROJECT_ID}
echo ${UPSTREAM_JOB_NAME}
echo ${UPSTREAM_PIPELINE_ID}
ls -al
curl -v -k --fail \
--header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
-o pipeline_jobs.json \
"${GITLAB_URL}/api/v4/projects/${UPSTREAM_PROJECT_ID}/pipelines/${UPSTREAM_PIPELINE_ID}/jobs?scope=success"
jq . pipeline_jobs.json
UPSTREAM_JOB_ID=$(jq --arg j_name "${UPSTREAM_JOB_NAME}" '.[] | select(.name==$j_name)|.id' pipeline_jobs.json)
# Download the artifacts for the job ID.
curl -v -k --fail \
--header "JOB-TOKEN: ${CI_JOB_TOKEN}" -o artifacts_test.zip \
"${GITLAB_URL}/api/v4/projects/${UPSTREAM_PROJECT_ID}/jobs/${UPSTREAM_JOB_ID}/artifacts"
ls -al
```
### Example Project
### What is the current *bug* behavior?
401 or gets the artifact of a previous successful run.
### What is the expected *correct* behavior?
Get the artifact from the upstream pipeline if the job with the artifact has succeeded but the pipeline is still pending.
### Relevant logs and/or screenshots
### Output of checks
Tested on 11.4
#### Results of GitLab environment info
#### Results of GitLab application Check
### Possible fixes
issue