Flexible job dependencies additional to simple stage dependencies

Problem to solve

we have many jobs, which are long running and blocking the pipeline, and increasing the overall time, because they are blocking the next stages.

Additional to stages, it would be great to have a job dependency system.

It would be great, if you can define also multiple dependencies to previous jobs.

Further details

Our current state is this

Build --|-- Package     --|-- Test-artifacts     --|-- Semaphore --|-- Deploy-dev --|-- Test-dev

maven ----- docker      ----- container_scanner  ----- semaphore ----- deploy-dev ----- test-dev
helm  -/ \- helm-upload -/ \- code_quality       -/
                           \- dependency_scanner -/
                           \- sast               -/
                           \- licence_managment  -/

Current duration is:

  • Job: 23m (Build + Package + Test-artifact + Semaphore + Deploy-dev + test-dev)
  • Build: 3m (max of one following jobs)
    • helm: 1m
    • maven: 3m
  • Package: 2m (max of one following jobs)
    • docker: 2m
    • helm-upload: 1m
  • Test-artifacts: 12m (max of one following jobs)
    • container_scanner: 5m
    • code_quality: 12m
    • dependency_scanner: 7m
    • sast: 5m
    • licence_managment: 7m
  • Semaphore: 1m (max of one following jobs)
    • semaphore: 1m
  • Deploy-dev: 2m (max of one following jobs)
    • deploy-dev: 2m
  • Test-dev: 3m (max of one following jobs)
    • test-dev: 3m

Proposal

Instead of that we could speed our pipeline but just using dependencies:

helm  ----- helm-upload -------------\
maven ----- docker  ----- semaphore ----- deploy-dev ----- test-dev
         |             \- container_scanner.........
         \- code_quality.........
         \- dependency_scanner.........
         \- sast.........   
         \- licence_managment.........

Expected duration is

  • Job: 15m (maven + code_quality)
    • helm + helm-upload + deploy-dev + test-dev: 7m
    • maven + docker + sempaphore + deploy-dev + test-dev: 11m
    • maven + docker + container_scanner: 10m
    • maven + code_quality: 15m
    • maven + dependency_scanner: 10m
    • maven + sast: 8m
    • maven + licence_managment: 10m

That would improve the parallelization, reduce the waiting time of unnecessary dependencies and speed up the whole pipeline. In our case, it could decrease the pipeline duration almost by 50%: 15m instead of 23m

What does success look like, and how can we measure that?

  • The number of parallelization should be increased
  • The duration of job should be decreased

Links / references

Edited by Michael Decker