Increase project:needs limit
Proposal
needs:project is limited to five projects.
Use needs:project to download artifacts from up to five jobs in other pipelines.
I need up to 30 projects. As a workaround i've created sub-jobs with each five needs:projects
entries and added them as dependencies of the main build job.
The limit could be changed by configuration or set to an higher default value like 50
Current Woraround
.artifacts.yml
.artifacts-collector:
stage: prepare
variables:
GIT_STRATEGY: none
script:
- (Dir -Recurse .\build\ | Get-Childitem).FullName
artifacts:
expire_in: 2h
paths:
- build/*
artifacts-collector-1:
extends: .artifacts-collector
needs:
- project: my-project/a
job: build
ref: main
artifacts: true
- project: my-project/b
job: build
ref: main
artifacts: true
- project: my-project/c
job: build
ref: main
artifacts: true
- project: my-project/d
job: build
ref: main
artifacts: true
- project: my-project/e
job: build
ref: main
artifacts: true
artifacts-collector-2:
extends: .artifacts-collector
needs:
- project: my-project/f
job: build
ref: main
artifacts: true
- project: my-project/g
job: build
ref: main
artifacts: true
- project: my-project/h
job: build
ref: main
artifacts: true
- project: my-project/i
job: build
ref: main
artifacts: true
- project: my-project/j
job: build
ref: main
artifacts: true
.gitlab-ci.yml
build:D28:
stage: build
dependencies:
- artifacts-collector-1
- artifacts-collector-2
- artifacts-collector-3
- artifacts-collector-4
- artifacts-collector-5
- artifacts-collector-6
- artifacts-collector-7
This workaround is not performant, because the artifcats must be processed twice (artifacts-collector-x
-jobs and in the build
-job). Another issue is that FF_ENABLE_JOB_CLEANUP=True
wont work with this jobs, because GIT_STRATEGY: none
. This ends up in increasing artifacts sizes. This happens because the successive collector jobs contain the artefacts of the previous jobs, as the artefacts are not cleaned up.
Order of investigation table
Group | Issue Link |
---|---|
spike | #413530 |
backend |
|