Flexible job dependencies additional to simple stage dependencies
Problem to solve
we have many jobs, which are long running and blocking the pipeline, and increasing the overall time, because they are blocking the next stages.
Additional to stages, it would be great to have a job dependency system.
It would be great, if you can define also multiple dependencies to previous jobs.
Further details
Our current state is this
Build --|-- Package --|-- Test-artifacts --|-- Semaphore --|-- Deploy-dev --|-- Test-dev
maven ----- docker ----- container_scanner ----- semaphore ----- deploy-dev ----- test-dev
helm -/ \- helm-upload -/ \- code_quality -/
\- dependency_scanner -/
\- sast -/
\- licence_managment -/
Current duration is:
- Job: 23m (Build + Package + Test-artifact + Semaphore + Deploy-dev + test-dev)
- Build: 3m (max of one following jobs)
- helm: 1m
- maven: 3m
- Package: 2m (max of one following jobs)
- docker: 2m
- helm-upload: 1m
- Test-artifacts: 12m (max of one following jobs)
- container_scanner: 5m
- code_quality: 12m
- dependency_scanner: 7m
- sast: 5m
- licence_managment: 7m
- Semaphore: 1m (max of one following jobs)
- semaphore: 1m
- Deploy-dev: 2m (max of one following jobs)
- deploy-dev: 2m
- Test-dev: 3m (max of one following jobs)
- test-dev: 3m
Proposal
Instead of that we could speed our pipeline but just using dependencies:
helm ----- helm-upload -------------\
maven ----- docker ----- semaphore ----- deploy-dev ----- test-dev
| \- container_scanner.........
\- code_quality.........
\- dependency_scanner.........
\- sast.........
\- licence_managment.........
Expected duration is
- Job: 15m (maven + code_quality)
- helm + helm-upload + deploy-dev + test-dev: 7m
- maven + docker + sempaphore + deploy-dev + test-dev: 11m
- maven + docker + container_scanner: 10m
- maven + code_quality: 15m
- maven + dependency_scanner: 10m
- maven + sast: 8m
- maven + licence_managment: 10m
That would improve the parallelization, reduce the waiting time of unnecessary dependencies and speed up the whole pipeline. In our case, it could decrease the pipeline duration almost by 50%: 15m instead of 23m
What does success look like, and how can we measure that?
- The number of parallelization should be increased
- The duration of job should be decreased
Links / references
- There was a similar (not same) request once: https://gitlab.com/gitlab-org/gitlab-ce/issues/26795
Edited by Michael Decker