Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Snippets
  • Sign up now
  • Login
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 46,530
    • Issues 46,530
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,505
    • Merge requests 1,505
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Artifacts
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • GitLabGitLab
  • Issues
  • #255983
Closed
Open
Issue created Sep 25, 2020 by Gianni@Gvazzana

'dotenv: report' artifacts replaced by newly run pipelines on the same commit ref

Summary

Unable to run pipelines off of the same commit, with out having the dotenv report artifact be replaced by the most recent run pipeline. Previous run pipelines download incorrect dotenv artifacts via their Needs: jobs.

Steps to reproduce

Example Project

What is the current bug behavior?

A dotenv report job is created in a Parent Pipeline (Pipeline A). to pass variables to downstream (child) pipelines within the same project.

In the downstream (Child) pipelines, the dotenv job is accessed via a 'Needs:' job for the project/job name/ref (branch)

If a new pipeline (Pipeline B) is run from the same commit, naturally it will create a dotenv Job with the same Job Name, from that same commit, in its own pipeline.

The issue is, Pipeline A's dotenv artifact has now been replaced by Pipeline B's artifact Any in progress / future pipeline A jobs will now download the wrong artifact via its Needs: job, and will fail due to invalid variables expected from Pipeline A's dotenv job.

What is the expected correct behavior?

Pipeline A creates a dotenv report job. Downstream pipelines (child pipelines) downloads the dotenv artifact via a Needs: job, that is tied to the $CI_PIPELINE_ID, not the Ref (branch or commit).

This would allow multiple pipelines to be run from the same project / branch / commit, with different variables passed in the dotenv report.

Current:

upstreamVariables:
  stage: createVariables
  environment:
    name: $CI_COMMIT_BRANCH
  tags:
    - win_pwsh
  needs:
  - project: $CI_PROJECT_PATH
    job: downstreamVariables
    ref: $CI_COMMIT_BRANCH
    artifacts: true
  script:
    - write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
  artifacts:
    reports:
      dotenv: dotenv.txt

Proposed:

upstreamVariables:
  stage: createVariables
  environment:
    name: $CI_COMMIT_BRANCH
  tags:
    - win_pwsh
  needs:
  - project: $CI_PROJECT_PATH
    job: downstreamVariables
    ref: **_$CI_PIPELINE_ID_**
    artifacts: true
  script:
    - write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
  artifacts:
    reports:
      dotenv: dotenv.txt

With the 'needs' job depending on the 'ref', it will pull the existing dotenv job artifact it finds on that commit, with the specified job name, rather then the job name for the intended Pipeline.

This would allow multiple dotenv report artifacts to exist, 1 for each pipeline, not one for the entire project / branch / commit, that when pulled, determines what variables you get (Artifact)

Possible fixes

Proposed:

upstreamVariables:
  stage: createVariables
  environment:
    name: $CI_COMMIT_BRANCH
  tags:
    - win_pwsh
  needs:
  - project: $CI_PROJECT_PATH
    job: downstreamVariables
    ref: **_$CI_PIPELINE_ID_**
    artifacts: true
  script:
    - write-host "This Deployment was triggered via '$CI_PARENT_SOURCE', from the '$CI_COMMIT_BRANCH' branch, and is using the '.upstreamVariables' job" `n -ForegroundColor Green
  artifacts:
    reports:
      dotenv: dotenv.txt

Proposal

To narrow down the scope of looking for artifacts for a specific job name we can introduce a new needs:pipeline syntax that allows downloading artifacts from a given pipeline ID.

build_job:
  needs:
    - pipeline: $CI_UPSTREAM_PIPELINE_ID
      job: $DEPENDENCY_JOB_NAME
      artifacts: true 

Upstream pipelines can pass their ID via trigger:variables in the bridge job and the downstream pipeline can use that ID to reference the right version of the artifact.

Edited Nov 17, 2020 by Fabio Pitino
Assignee
Assign to
Time tracking