'dotenv: report' artifacts replaced by newly run pipelines on the same commit ref
Summary
Unable to run pipelines off of the same commit, with out having the dotenv report artifact be replaced by the most recent run pipeline. Previous run pipelines download incorrect dotenv artifacts via their Needs: jobs.
Steps to reproduce
Example Project
What is the current bug behavior?
A dotenv report job is created in a Parent Pipeline (Pipeline A). to pass variables to downstream (child) pipelines within the same project.
In the downstream (Child) pipelines, the dotenv job is accessed via a 'Needs:' job for the project/job name/ref (branch)
If a new pipeline (Pipeline B) is run from the same commit, naturally it will create a dotenv Job with the same Job Name, from that same commit, in its own pipeline.
The issue is, Pipeline A's dotenv artifact has now been replaced by Pipeline B's artifact Any in progress / future pipeline A jobs will now download the wrong artifact via its Needs: job, and will fail due to invalid variables expected from Pipeline A's dotenv job.
What is the expected correct behavior?
Pipeline A creates a dotenv report job. Downstream pipelines (child pipelines) downloads the dotenv artifact via a Needs: job, that is tied to the $CI_PIPELINE_ID, not the Ref (branch or commit).
This would allow multiple pipelines to be run from the same project / branch / commit, with different variables passed in the dotenv report.
Current:
upstreamVariables:
stage: createVariables
environment:
name: $CI_COMMIT_BRANCH
tags:
- win_pwsh
needs:
- project: $CI_PROJECT_PATH
job: downstreamVariables
ref: $CI_COMMIT_BRANCH
artifacts: true
script:
- write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
artifacts:
reports:
dotenv: dotenv.txt
Proposed:
upstreamVariables:
stage: createVariables
environment:
name: $CI_COMMIT_BRANCH
tags:
- win_pwsh
needs:
- project: $CI_PROJECT_PATH
job: downstreamVariables
ref: **_$CI_PIPELINE_ID_**
artifacts: true
script:
- write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
artifacts:
reports:
dotenv: dotenv.txt
With the 'needs' job depending on the 'ref', it will pull the existing dotenv job artifact it finds on that commit, with the specified job name, rather then the job name for the intended Pipeline.
This would allow multiple dotenv report artifacts to exist, 1 for each pipeline, not one for the entire project / branch / commit, that when pulled, determines what variables you get (Artifact)
Possible fixes
Proposed:
upstreamVariables:
stage: createVariables
environment:
name: $CI_COMMIT_BRANCH
tags:
- win_pwsh
needs:
- project: $CI_PROJECT_PATH
job: downstreamVariables
ref: **_$CI_PIPELINE_ID_**
artifacts: true
script:
- write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
artifacts:
reports:
dotenv: dotenv.txt
Proposal
To narrow down the scope of looking for artifacts for a specific job name we can introduce a new needs:pipeline
syntax that allows downloading artifacts from a given pipeline ID.
build_job:
needs:
- pipeline: $CI_UPSTREAM_PIPELINE_ID
job: $DEPENDENCY_JOB_NAME
artifacts: true
Upstream pipelines can pass their ID via trigger:variables
in the bridge job and the downstream pipeline can use that ID to reference the right version of the artifact.