Skip to content

GitLab Next

Open
Created by Fabio Pitino@fabiopitino🔴Maintainer

Address conflicting syntax `needs:pipeline`

Background

In #255983 (comment 444777769) we had a customer requirement to download artifacts from a parent pipeline and not from the latest successful pipeline for a given ref.

To solve this problem we thought of adding a new syntax:

download-artifacts:
  needs:
    - job: generate-artifacts
      pipeline: $CI_PARENT_PIPELINE_ID
      artifacts: true

This syntax is in line with the principles of cross project artifacts download and artifacts download via needs within the same pipeline. The difference is that allowing to fetch artifacts from a job in a specific pipeline narrows down the scope and removes potential conflicts with other running pipelines (e.g. same project and same ref).

However, a similar syntax is available today to mirror the status of an upstream pipeline:

upstream-status:
  needs:
    - pipeline: some-group/another-project

This syntax turns upstream-status into a bridge job, opposite of trigger keyword, and mirrors the status of an upstream pipeline.

Problem

Introducing the new syntax side-by-side is possible but it can cause confusion because the differences are too subtle:

# This is considered as a Job that downloads artifacts from a parent pipeline's `build` job
rspec:
  script: rspec
  needs:
    - pipeline: $CI_PARENT_PIPELINE_ID
      job: build
      artifacts: true

# This is considered as a Bridge that mirrors the status of `upstream-project` latest pipeline on `master`
upstream-status:
  needs:
    - pipeline: some-group/upstream-project

Additional notes

I think needs keyword is already overloaded of responsibilities:

  • allows to define the sequence of jobs (DAG)
  • allows to download artifacts from a previous job in the same pipeline
  • allows to download artifacts from a job in the same project but different ref
  • allows to download artifacts from a job in the same project and same ref
  • allows to download artifacts from a job in another project and ref
  • allows to mirror the status of an upstream project pipeline

But maybe combining both DAG and artifacts download with needs is acceptable. However, mirroring the upstream pipeline status could have been done with a different syntax

Proposal

We could introduce a new syntax for upstream status mirroring that doesn't use needs. Example:

upstream-status:
  reflect: some-group/upstream-project

This way we could distinguish between bridge and job better. A bridge defines either trigger or reflect, while a job defines script. Instead the current code allows needs to be used in Job for dependency management and in Bridge for status mirroring.

Then, we could translate automatically configs in the form needs:pipeline (without job) into reflect: syntax so it's backward compatible with existing syntax. This allows us to introduce a new syntax while supporting the old one. In %14.0 we could deprecate needs:pipeline for status mirroring.

cc @matteeyah @mbobin @ayufan @grzesiek

Edited by Fabio Pitino