Increase project:needs limit

Proposal

needs:project is limited to five projects.

Use needs:project to download artifacts from up to five jobs in other pipelines.

I need up to 30 projects. As a workaround i've created sub-jobs with each five needs:projects entries and added them as dependencies of the main build job.

The limit could be changed by configuration or set to an higher default value like 50

Current Woraround

.artifacts.yml

.artifacts-collector:
  stage: prepare
  variables:
    GIT_STRATEGY: none
  script:
    - (Dir -Recurse .\build\ | Get-Childitem).FullName
  artifacts:    
    expire_in: 2h
    paths: 
      - build/*

artifacts-collector-1:
  extends: .artifacts-collector
  needs:
    - project: my-project/a
      job: build
      ref: main
      artifacts: true
    - project: my-project/b
      job: build
      ref: main
      artifacts: true
    - project: my-project/c
      job: build
      ref: main
      artifacts: true
    - project: my-project/d
      job: build
      ref: main
      artifacts: true
    - project: my-project/e
      job: build
      ref: main
      artifacts: true

artifacts-collector-2:
  extends: .artifacts-collector
  needs:
    - project: my-project/f
      job: build
      ref: main
      artifacts: true
    - project: my-project/g
      job: build
      ref: main
      artifacts: true
    - project: my-project/h
      job: build
      ref: main
      artifacts: true
    - project: my-project/i
      job: build
      ref: main
      artifacts: true
    - project: my-project/j
      job: build
      ref: main
      artifacts: true

.gitlab-ci.yml

build:D28:
  stage: build
  dependencies:
    - artifacts-collector-1
    - artifacts-collector-2
    - artifacts-collector-3
    - artifacts-collector-4
    - artifacts-collector-5
    - artifacts-collector-6
    - artifacts-collector-7

This workaround is not performant, because the artifcats must be processed twice (artifacts-collector-x-jobs and in the build-job). Another issue is that FF_ENABLE_JOB_CLEANUP=True wont work with this jobs, because GIT_STRATEGY: none. This ends up in increasing artifacts sizes. This happens because the successive collector jobs contain the artefacts of the previous jobs, as the artefacts are not cleaned up.

Order of investigation table

Group	Issue Link
spike	#413530
backend	👈 You are here