Difficulty using needs when extending from job with dependencies

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

In the documentation for gitlab-ci both needs and dependencies, both say "you should not combine dependencies with needs in the same job." That's good advice, the behavior isn't documented or guarenteed to be stable. The problem is if you're extending from a job with one and need to use the other. For instance, the job you depends on has dependencies: [], but you have several optional needs (e.g. things from optional jobs that create dotenv reports).

Proposal

There should be a way to explicitly or implicitly override a needs or dependencies field when extending. Some ideas:

  1. Add a wildcard which would set you back to default behavior (e.g. dependencies: * or dependencies: ['*'] is the same as when the field is omitted)
  2. If one is specified by an inheriting or extending job, ignore both from upstream (i.e. if needs is specified, ignore both needs and dependencies from base jobs)
  3. Same as 2, but have it be well defined if both are in the same structure (e.g. ignore dependencies if needs is specified at all.)
  4. An explicit field to ignore the other (e.g. ignore_dependencies = true)
  5. Have one always overwrite the other (or error).. e.g. if needs is set, ignore dependencies.
  6. Have artifacts pulled from the union of needs and dependencies, rather than the intersection.

My favorite is 1. I do think 2 or 6 is most intuitive, but also not strictly backwards compatible.

Confirm purpose and User Reception (how does this benefit the user?)

This is most useful for pipeline authors who have a base that they don't control, which defines needs or dependencies. Perhaps this could effect other things, but those, in particular, don't have anything you can set that would be the same as the omitted behavior.

Additional details

Below is a specific pipeline which demonstrates the issue:

.gitlab-ci-base.yml

run_job:
  script:
   - export
  dependencies: []

.gitlab-ci.yml

include:
  - local: .gitlab-ci-base.yml

a:
  script:
    - |
      cat <<EOF > a.env
      JOB_A_VARIABLE=value
      EOF
  artifacts:
    reports:
      dotenv: a.env
      
b:
  script:
    - |
      cat <<EOF > b.env
      JOB_B_VARIABLE=value
      EOF
  artifacts:
    reports:
      dotenv: b.env


run_job:
  needs:
    - a
    - job: b
      optional: true
    - job: c
      optional: true

This will result in neither JOB_A_VARIABLE or JOB_B_VARIABLE being exported. You can add a to dependencies, but there isn't a way to have b be optionally included.

Links/References

Edited by 🤖 GitLab Bot 🤖