Skip to content

'dotenv: report' artifacts replaced by newly run pipelines on the same commit ref

Summary

Unable to run pipelines off of the same commit, with out having the dotenv report artifact be replaced by the most recent run pipeline. Previous run pipelines download incorrect dotenv artifacts via their Needs: jobs.

Steps to reproduce

Example Project

What is the current bug behavior?

A dotenv report job is created in a Parent Pipeline (Pipeline A). to pass variables to downstream (child) pipelines within the same project.

In the downstream (Child) pipelines, the dotenv job is accessed via a 'Needs:' job for the project/job name/ref (branch)

If a new pipeline (Pipeline B) is run from the same commit, naturally it will create a dotenv Job with the same Job Name, from that same commit, in its own pipeline.

The issue is, Pipeline A's dotenv artifact has now been replaced by Pipeline B's artifact Any in progress / future pipeline A jobs will now download the wrong artifact via its Needs: job, and will fail due to invalid variables expected from Pipeline A's dotenv job.

What is the expected correct behavior?

Pipeline A creates a dotenv report job. Downstream pipelines (child pipelines) downloads the dotenv artifact via a Needs: job, that is tied to the $CI_PIPELINE_ID, not the Ref (branch or commit).

This would allow multiple pipelines to be run from the same project / branch / commit, with different variables passed in the dotenv report.

Current:

upstreamVariables:
  stage: createVariables
  environment:
    name: $CI_COMMIT_BRANCH
  tags:
    - win_pwsh
  needs:
  - project: $CI_PROJECT_PATH
    job: downstreamVariables
    ref: $CI_COMMIT_BRANCH
    artifacts: true
  script:
    - write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
  artifacts:
    reports:
      dotenv: dotenv.txt

Proposed:

upstreamVariables:
  stage: createVariables
  environment:
    name: $CI_COMMIT_BRANCH
  tags:
    - win_pwsh
  needs:
  - project: $CI_PROJECT_PATH
    job: downstreamVariables
    ref: **_$CI_PIPELINE_ID_**
    artifacts: true
  script:
    - write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
  artifacts:
    reports:
      dotenv: dotenv.txt

With the 'needs' job depending on the 'ref', it will pull the existing dotenv job artifact it finds on that commit, with the specified job name, rather then the job name for the intended Pipeline.

This would allow multiple dotenv report artifacts to exist, 1 for each pipeline, not one for the entire project / branch / commit, that when pulled, determines what variables you get (Artifact)

Possible fixes

Proposed:

upstreamVariables:
  stage: createVariables
  environment:
    name: $CI_COMMIT_BRANCH
  tags:
    - win_pwsh
  needs:
  - project: $CI_PROJECT_PATH
    job: downstreamVariables
    ref: **_$CI_PIPELINE_ID_**
    artifacts: true
  script:
    - write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
  artifacts:
    reports:
      dotenv: dotenv.txt

Proposal

To narrow down the scope of looking for artifacts for a specific job name we can introduce a new needs:pipeline syntax that allows downloading artifacts from a given pipeline ID.

build_job:
  needs:
    - pipeline: $CI_UPSTREAM_PIPELINE_ID
      job: $DEPENDENCY_JOB_NAME
      artifacts: true

Upstream pipelines can pass their ID via trigger:variables in the bridge job and the downstream pipeline can use that ID to reference the right version of the artifact.

Edited by Fabio Pitino