[matrix job] allow customization of resulting job name

Problem

parallel:matrix generates job names as job: [value1, value2, ...], using the full value of every matrix variable. When values are long (image references, URLs, paths) or there are many variables:

  • Job names become hard to read in the pipeline graph and job lists. Users have to hover over each job to tell them apart.
  • Generated names can exceed the 255-character job name limit, so the pipeline fails to be created, even when each part on its own is short enough.

This issue tracks letting users control the generated job name. The work is split into iterations.

Iteration 1: value aliases (in progress: !258125)

Matrix values accept an optional { value:, alias: } form. The alias appears in the generated job name, and the CI/CD variable keeps the full value. Design agreed in this comment.

build:
  parallel:
    matrix:
      - OS: [ubuntu, alpine]
        IMAGE:
          - value: registry.example.com/builder-u20:0.0.3
            alias: u20
          - registry.example.com/builder-u22:0.0.1

Generates build: [ubuntu, u20], build: [ubuntu, registry.example.com/builder-u22:0.0.1], and so on. $IMAGE always contains the full value.

  • Aliased and plain values can be mixed in the same matrix.
  • needs:parallel:matrix can match a job by its full value or its alias. The value/alias hash form itself is not accepted in needs.
  • When aliases are used, matrices that would generate duplicate job names are rejected. The 255-character limit still applies.
  • Matrices without aliases keep the current behavior, so this is not a breaking change.
  • Behind the ci_parallel_matrix_value_alias feature flag (beta, off by default).

Iteration 2: custom job name templates (not yet scoped)

The original request: let users define the whole generated job name, not just shorten individual values. Related requests from the comments:

  • Leave specific matrix variables out of the generated name. For example, a variable needed by one job but not by the jobs that needs it.
  • Fully custom names, for when jobs share a long base name and differ by only one value.
Original proposal (2020)

It may be interesting to allow customization of resulting job name with parallel:matrix. By exposing a variable CI_NODE_PARENT_NAME, we could let user specify the jobs name. For example

deploystacks:
   parallel:
     resulting_job_name: "%CI_NODE_PARENT_NAME - %PROVIDER - %STACK"
     matrix:
      - PROVIDER: aws
        STACK:
          - monitoring
          - app1
          - app2
      - PROVIDER: ovh
        STACK: [monitoring, backup, app]
      - PROVIDER: [gcp, vultr]
        STACK: [data, processing]

This will generate job names

  • deploystacks - aws - monitoring
  • deploystacks - aws - app1
  • deploystacks - aws - app2
  • deploystacks - ovh - monitoring ...

The idea is to lower the length that can be annoying when using one dimensional matrices introduced in gitlab 13.5 and let user only expose the value of the matrix element instead of parent-node-name [ matrix item]. This behaviour leads to a troublesome experience as, to find the good job, you have to scroll and overlay with the mouse to find the right job you want to access.

Note (2024): the example names above are no shorter than the default format (deploystacks: [aws, monitoring]). The main value of templates is control over the name, not length.

Open questions before iteration 2 can be scheduled:

  • Syntax: a new %VAR format, or reuse the existing $[[ matrix.KEY ]] matrix expressions? Community feedback leans toward matrix expressions.
  • What happens when a template leaves out a matrix key: an error, an empty value, or a numbered fallback?
  • Is a new CI_NODE_PARENT_NAME variable needed, given that CI_JOB_GROUP_NAME already exists?
  • How do we keep generated names unique, and how does needs:parallel:matrix match jobs with custom names?
  • Do custom names break job grouping in the pipeline graph?
  • Can a UI improvement for long job names cover some of these cases instead? See #468449.

Rollout

  • Iteration 1: !258125 merged (milestone: TBD)
  • Iteration 1: rollout issue for ci_parallel_matrix_value_alias
  • Iteration 1: feature flag on by default, then GA
  • Iteration 1: release post item, and notify the CSMs who logged customer interest on this issue
  • Iteration 2: open questions answered and issue refined
Edited by 🤖 GitLab Bot 🤖