### Problem to Solve We recently introduced child/parent pipelines (https://gitlab.com/gitlab-org/gitlab/-/issues/16094) which allows for writing your own code to generate a child/parent pipeline - this is a powerful way to generate any custom behaviors, but it is a bit heavyweight for simpler scenarios where you just want to create jobs for a simple matrix. ### Proposal We will add a `matrix` keyword to `parallel`, which is a pre-existing keyword that handles parallelization of jobs: ```yaml deploystacks: stage: deploy parallel: matrix: - PROVIDER: aws STACK: [monitoring, app1, app2] - PROVIDER: ovh STACK: [monitoring, backup, app] - PROVIDER: gcp STACK: [data, processing] ``` The above would generate the following parallel jobs: 1. `deploystacks (PROVIDER=aws; STACK=monitoring)` 1. `deploystacks (PROVIDER=aws; STACK=app1)` 1. `deploystacks (PROVIDER=aws; STACK=app2)` 1. `deploystacks (PROVIDER=ovh; STACK=monitoring)` 1. `deploystacks (PROVIDER=ovh; STACK=backup)` 1. `deploystacks (PROVIDER=ovh; STACK=app)` 1. `deploystacks (PROVIDER=gcp; STACK=data)` 1. `deploystacks (PROVIDER=gcp; STACK=processing)` The benefits here are: - This keeps the `matrix` keyword to avoid the issue of being unable to extend it in a reasonable way later: https://gitlab.com/gitlab-org/gitlab/-/issues/15356#note_309409940. Yes, this is a little less concise than not having the keyword, but it makes the intention very clear. - More complex scenarios can still use child/parent pipelines, so we don't have to design this as a all inclusive matrix build specification language. Additionally, a simpler matrix is also supported for when each entry from the first dimension contains all items from the second: ```yaml deploystacks: stage: deploy parallel: matrix: - PROVIDER: [aws, ovh, gcp] STACK: [monitoring, app1, app2] ``` This version would generate: 1. `deploystacks (PROVIDER=aws; STACK=monitoring)` 1. `deploystacks (PROVIDER=aws; STACK=app1)` 1. `deploystacks (PROVIDER=aws; STACK=app2)` 1. `deploystacks (PROVIDER=ovh; STACK=monitoring)` 1. `deploystacks (PROVIDER=ovh; STACK=app1)` 1. `deploystacks (PROVIDER=ovh; STACK=app2)` 1. `deploystacks (PROVIDER=gcp; STACK=monitoring)` 1. `deploystacks (PROVIDER=gcp; STACK=app1)` 1. `deploystacks (PROVIDER=gcp; STACK=app2)` ### Limitations - For the MVC total jobs will be limited to the same number as `parallel` (currently 50: https://docs.gitlab.com/ee/ci/yaml/#parallel) - There is no such thing as a variable containing an array in GitLab today, so setting up a matrix as follows is not possible: ```yaml parallel: matrix: - PROVIDER: ${MY_PROVIDERS} STACK: ${MY_STACKS} ``` Instead of this, for now to avoid repitition you should use `extends`: ```yaml .matrix parallel: matrix: - PROVIDER: [aws, ovh, gcp] STACK: [monitoring, app1, app2] build: extends: .matrix test: extends: .matrix deploy: extends: .matrix ``` ### External examples Example of `.travis.yml` ```yaml rvm: - 1.9.3 - 2.0.0 - 2.2 - ruby-head - jruby - rbx-2 - ree gemfile: - gemfiles/Gemfile.rails-2.3.x - gemfiles/Gemfile.rails-3.0.x - gemfiles/Gemfile.rails-3.1.x - gemfiles/Gemfile.rails-edge env: - ISOLATED=true - ISOLATED=false ``` This produces 56 individual jobs. Real life example provided by @MrChrisW: * https://gitlab.com/MrChrisW/cookiecutter-pylibrary/blob/master/.travis.yml * https://gitlab.com/MrChrisW/cookiecutter-pylibrary/blob/master/.gitlab-ci.yml Travis docs: [Build matrix](https://docs.travis-ci.com/user/customizing-the-build/#build-matrix) Also, GitHub CI/CD is planning the same feature: https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategy ### Related Links Related issues https://gitlab.com/gitlab-org/gitlab-ce/issues/13755, https://gitlab.com/gitlab-org/gitlab-ce/issues/19198 ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> * Unit test changes - Yes * Ensure [parallel](https://docs.gitlab.com/ee/ci/yaml/README.html#parallel) existing limitations are preserved: required minimum 2 jobs and maximum 50 job instances, and empty arrays are not allowed. * Ensure the appropriate job instances are created as specified in matrix * Ensure multiple jobs can `extends` the same matrix and job instances count is accordingly. * Integration test changes - not required * End-to-end test change - not required Please see the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning /cc @MrChrisW @markpundsack @ayufan