Skip to content

Allow artifacts download from specific pipeline ID

Fabio Pitino requested to merge artifact-download-from-pipeline-id-with-needs into master

What does this MR do?

Related to #255983 (closed).

Today some customers are using artifacts download feature on the same project via needs to accomplish artifacts passing from parent to child pipelines. While this might work, it can produce inconsistent results because we are downloading the artifacts from the latest job for the given ref. If there are multiple pipelines for the same ref running at the same time, one child pipeline might download artifacts produced by another parent pipeline.

This MR introduces a new syntax in the backend that allows automatic download of artifacts from a job in a given pipeline ID. This solution solves the problem of having artifacts being overridden by other pipelines on the same ref.

Suppose we have a parent pipeline (same would work also with multi-project pipelines):

build:
  script: ./build
  artifacts:
    paths: out/*.txt

trigger:
  include: child.yml
  variables:
    UPSTREAM_PIPELINE_ID: $CI_PIPELINE_ID

and a child pipeline:

rspec:
  stage: test
  script:
    - ls out/*.txt
    - ./rspec
  needs:
    - pipeline: $UPSTREAM_PIPELINE_ID
      job: build

This will allow the job rspec in the downstream pipeline to require a download of artifacts from build in a specific pipeline (by ID).

Note that this is a backend-only change and won't have any effect now. We are only allowing the syntax to store the cross pipeline dependencies but we are not yet using them.

As part of this MR I also renamed the existing cross_pipeline methods to cross_project because they were related to dependencies specified by project + ref.

In a follow-up MR I will add support for Rails to actually manage the cross pipeline dependencies. In that MR I will also add a changelog and documentation.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Grzegorz Bizon

Merge request reports