Resolve "Backend: Allow needs to use the parallel keyword"

What does this MR do and why?

This MR adds support for parallel:matrix for the needs keyword.

As mentioned in the docs, currently if needs refers to a job that uses the parallel keyword, it depends on all jobs created in parallel, not just one job.

To visualize this, we currently have support for the following:

build:
  parallel:
    matrix:
      - PLATFORM: ubuntu
        STACK: [python, ruby]

test:
  needs:
    - job: build

# The job `test` will run once all parallelized jobs from build (`build: [ubuntu, python]`, `build: [ubuntu, ruby]`) are completed

This MR works to add parallel:matrix to the needs keyword:

build:
  parallel:
    matrix:
      - PLATFORM: ubuntu
        STACK: [python, ruby]

test:
  needs:
    - job: build
      parallel:
        matrix:
          - PLATFORM: ubuntu
            STACK: python

# The job `test` will run once the job `build: [ubuntu, python]` is completed

Feature flag: ci_support_needs_parallel_matrix

Resolves #254821 (closed)

How to set up and validate locally

Testing with multiple concurrent runners

In the following video, please observe that job5 gets queued and completes after job1: [p2, s1] finishes:

Screen_Recording_2023-07-17_at_12.10.39_AM

1. Please ensure that you have 3 concurrent runners running locally. This can be done by:

• Registering 3 runners
• Under your gdk.yml file, please set concurrent: 3 under runner as well as the 3 tokens:

runner:
  concurrent: 3
  enabled: true
  executer: docker
  install_mode: docker
  token: TOKEN_1 
  token: TOKEN_2
  token: TOKEN_3

2. Create a project with the following .gitlab-ci.yml file:

yml file

job1:
  stage: build
  script:
    - echo "First job for build stage"
    - sleep 10
  parallel:
    matrix:
      - PROVIDER: [p1, p2]
        STACK: [s1, s2]

job2:
  stage: build
  script:
    - echo "Second job for build stage"
    - sleep 10

job3:
  stage: build
  script:
    - echo "Third job for build stage"
  needs: [job2]

job4:
  stage: test
  script:
    - echo "First job for test stage"
  needs: [job1]

job5:
  stage: test
  script:
    - echo "Second job for test stage"
  needs:
    - job: job1
      parallel:
        matrix:
          - PROVIDER: [p2]
            STACK: [s1]

3. Turn on the Feature Flag in your local environment: Feature.enable(:ci_support_needs_parallel_matrix).
4. Run the pipeline.
5. Observe that the job job5 gets queued and started after job1: [p2, s1] is complete.
6. Observe that the job job5 completes before job1 and job2, even though job2 is from an earlier stage and job1 is "needed" by job5

Screenshots of `job5` completing after `job1: [p2, s1]` finishes

Testing with one runner

In the following video, please observe that job4 gets queued as soon as job1: [p1, s1] has completed:

Screen_Recording_2023-07-17_at_12.26.56_AM

1. Create a project with the following .gitlab-ci.yml file:

yml file

job1:
  stage: build
  script:
    - echo "First job for build stage"
  parallel:
    matrix:
      - PROVIDER: [p1, p2]
        STACK: [s1, s2]

job2:
  stage: build
  script:
    - echo "Second job for build stage"

job3:
  stage: build
  script:
    - echo "Third job for build stage"
  needs: [job2]

job4:
  stage: test
  script:
    - echo "First job for test stage"
  needs:
    - job: job1
      parallel:
        matrix:
          - PROVIDER: [p1]
            STACK: [s1]

2. Turn on the Feature Flag in your local environment: Feature.enable(:ci_support_needs_parallel_matrix).
3. Run the pipeline
4. Observe that job4 job gets queued after job1: [p1, s1] has completed. This should be queued earlier or as early as job3 even though the test stage comes after the build stage

Screenshots of the start of pipeline

Screenshots of when the parallelized job specified by `needs:parallel:matrix` is completed

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #254821 (closed)