Allow parallel jobs in cross-projects artifacts dependencies

Summary

!20721 (merged) introduces the ability to download artifacts by job name from other projects pipelines, but it doesn't work if the requested build is defined as a parallel job.

Steps to reproduce

CI file for project-a:

build-job:
  stage: build
  script:
    - mkdir -p features/
    - echo $(date) > features/file-$CI_NODE_INDEX.txt
  artifacts:
    paths:
      - features/      
  parallel: 2

CI file for project-b:

test-job:
  stage: test
  script:
    - ls -lhR
  needs:
    - project: project-a
      ref: master 
      job: build-job
      artifacts: true

What is the current bug behavior?

test-job does not download any artifacts from build-job.

What is the expected correct behavior?

test-job must download artifacts from all the build-job instances.

Possible fixes

In our project-b we're looking for one build-job when we should be looking for two jobs, named build-job 1/2 and build-job 2/2. The workaround would be to specify the 1/2 and 2/2 parts in the needs definition, but it tightly couples the two pipelines.

Because we change the job name when saving it to include the parallel data, we don't have an efficient way of searching for the build-job. We can do some pattern matching on the name, but we need to group them by something common(like the pipeline id) to get the relevant ones:

scope = Project.find_by_full_path('user/project-a').builds.latest.success
builds = scope.where(name: 'build-job').or(scope.where("name SIMILAR TO 'build-job [0-9]+/[0-9]+'"))
builds.where(commit_id: builds.select('max(commit_id)'))

But if one of the build-job builds fails, the pipeline from project-b would receive only 1 dependency.

We can turn this around and start from the successful pipelines to make sure that all the builds are successful:

project = Project.find_by_full_path('user/project-a')
pipeline = project.ci_pipelines.success.last
scope = pipeline.builds.success.latest
builds = scope.where(name: 'build-job').or(scope.where("name SIMILAR TO 'build-job [0-9]+/[0-9]+'"))

But the problem here is that this pipeline might not include the build-job because it was filtered out by rules. Parent/Child pipelines must also be considered when searching for this build.

We must also think about limits for this feature. Because we are determining the final count of dependencies during the runtime, we can exceed the imposed limit by 50 times(the limit for parallel is 50).