Support dynamic job dependencies in `needs:parallel:matrix` - Beta
## Update
This functionality is now available in GitLab 18.6 as a Beta feature. We're planning to promote it to General Availability (GA) in version 18.8.
We'd love your feedback: If you have any suggestions or encounter issues during the Beta period, please share your feedback in the [GA issue](https://gitlab.com/gitlab-org/gitlab/-/issues/581803)
Your input will help us ensure a smooth transition to GA.
## Summary
Enable variable expansion for matrix variables in `needs:parallel:matrix` to support dynamic job dependencies based on matrix values.
## Problem
Currently, when using `parallel:matrix` to create multiple jobs, you cannot dynamically reference the matrix variables in `needs:parallel:matrix`. This forces users to manually create separate jobs for each matrix permutation, which doesn't scale well.
## Validated Solution
Based on the PoC in !202964+, we will implement matrix expressions using the `$[[ matrix.VARIABLE_NAME ]]` syntax.
### Syntax
```yaml
unit-test:
stage: test
script:
- echo "Running tests for $OS-$ARCH-$ENV"
needs:
- job: build
parallel:
matrix:
- OS: $[[ matrix.OS ]]
ARCH: $[[ matrix.ARCH ]]
ENV: $[[ matrix.ENV ]]
parallel:
matrix:
- OS: ["ubuntu", "alpine", "centos"]
ARCH: ["amd64", "arm64"]
ENV: ["dev", "prod"]
```
### How it works
- Matrix expressions in `needs:parallel:matrix` use the `$[[ matrix.VARIABLE_NAME ]]` syntax to reference variables from the current job's matrix
- Each parallel job will resolve its own matrix values when determining dependencies
- This allows 1:1 mapping between matrix jobs across stages without manual duplication
### Why not just expand variables?
We're introducing the `$[[ matrix.VAR ]]` syntax instead of expanding `$VAR` because of our long-term initiative to provide a more structured and predictable system than variables.
1. **Improved UX**: Matrix variables are not regular CI/CD variables, and treating them as such would create confusion about variable precedence and scope. The CI expression `matrix` context makes it explicit that you're referencing matrix values, not environment variables or another type of CI/CD variable
2. **Future consistency**: Following the [job inputs](https://gitlab.com/groups/gitlab-org/-/epics/17833) work, users will be able to use `${{ matrix.VAR }}` in CI keywords evaluated by Runner (like `script`), where matrix variables are currently available
<details>
<summary>Original Description</summary>
I am aware of [255311](https://gitlab.com/gitlab-org/gitlab/-/issues/255311). Requested to be a separate issue by @dhershkovitch [here](https://gitlab.com/gitlab-org/gitlab/-/issues/255311#note_1531936752).
### Proposal
Expand variables that are added to the job via `parallel:matrix` when used in `needs:parallel:matrix`. This is very useful for pipelines with a matrix that creates N jobs in e.g. stage `build` and subsequently also needs _N_ jobs in a later stag e e.g. `test`, where each test-job depends on a specific build-job from the matrix. Consider the following example:
```yaml
.parallel-strat:
parallel:
matrix:
- TARGET: ["linux", "windows"]
build-job:
parallel: !reference [.parallel-strat, parallel]
# this would be great but it's not working, because `${TARGET}` is not expanded
test-job:
parallel: !reference [.parallel-strat, parallel]
needs:
- job: build-job
parallel:
matrix:
- TARGET: ${TARGET}
# status quo, create a dedicated job for every possible value of TARGET
test-job-linux:
needs:
- job: build-job
parallel:
matrix:
- TARGET: "linux"
test-job-windows:
needs:
- job: build-job
parallel:
matrix:
- TARGET: "windows"
```
As you can imagine, the current behavior does not scale well. The example above only considers two stages that need the N jobs and a single matrix variable. If there are multiple stages and/or multiple matrix variables, things get worse pretty f ast.
Currently, in a scenario where I have _N_ jobs and _M_ stages that all need the _N_ jobs, I can effectively reduce the _N\*M_ total jobs to _N\*(M-1)_ jobs + 1 matrix job (in the first stage). With variable expansion this number can be brought d own to _M_ matrix jobs.
</details>
issue