Support yaml sequences and maps in CI yaml variables

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem to solve

Supporting array variables in CI allows for

  1. Pipeline code to be more DRY and composed more easily - script jobs can expect an array input in a well known variable, and be reused in more locations. (Rather than having to implement parsing string variables into collections, and their many purpose)
  2. .gitlab-ci.yaml to be more compatible with plain yaml
  3. Relates to concepts such as parent/child pipelines and batching (https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-batching?view=vs-2019), allowing pipelines to operate almost polymorphically depending on their inputs or combinations of inputs (multi dispatch). This extends the reach of pipelines and automation

A use case

  • a variable supporting a list of servers to iterate over, applying IAC on each one. Different pipeline jobs would have a list of servers, for different environments.
  • a list of files to upload to different sites.

Intended users

Further details

Presently the following doesn't work - first fails at runtime, second fails parsing some_job_with_loop_over_array_variable: image: CI_REGISTRY/platform/images/alpine-multiuse:latest variables: Zips[0]: 'file1.zip' Zips[1]: 'file2.zip' Zips[2]: 'file3.zip' script: | for i in "{Zips[@]}"; do echo $i done

some_job_with_loop_over_array_variable2: image: CI_REGISTRY/platform/images/alpine-multiuse:latest variables: Zips: ['file1.zip', 'file2.zip', 'file3.zip'] script: | for i in "{Zips[@]}"; do echo $i done

Proposal

Parse variables using yaml parsing techniques, introspect for being a collection type, map to appropriate shell variables in the appropriate shell employed by the pipeline.

Permissions and Security

Prevent overflows, and DOS.

Documentation

Yes

Testing

On it's own, little additional complexity. However with each additional piece of functionality in the .gitlab-ci.yaml, if error reporting is not good, debugging gets increasingly difficult. e.g Use of variables in rules could be expanded to include "in" as an operator once collections are allowed, but this would add complexity to determining why a job did or didn't evaluate to be run in a pipeline.

What does success look like, and how can we measure that?

Gitlab being intrinsically motivated, and appreciative of beauty in code, the simple realisation of this feature will be evidence of success at the highest level.

What is the type of buyer?

The kind of buyer that can't stop at one variable... he or she needs a whole collection

Links / references

Edited by 🤖 GitLab Bot 🤖