Allow being more DRY in `.gitlab-ci.yml` by flattening arrays
TL;DR: the processing of gitlab-ci.yml
should automatically flatten all variables containing arrays (services
, script
, before_script
, ...).
Description
currently code snippets cannot be shared between multiple jobs/stages
, everything has to be duplicated, which is not DRY
test1:
script:
- default_step1
- default_step2
- job_test1_step1
- job_test1_step2
test2:
script:
- default_step1
- default_step2
- job_test2_step
- job_test2_step2
there is also the before_script
but often i want only to share code between some stages/jobs
and not all.
before_script:
- default_step1
- default_step2
test1:
script:
- job_test1_step1
- job_test1_step2
test2:
script:
- job_test2_step
- job_test2_step2
test3:
script:
# the before script will also be called here, where i don't want it to be called
- job_test3_step
- job_test2_step2
of course we have the yaml anchor feature but this won't help us for shared script code because the yaml parser will overwrite array sections and don't merge them. only hashes will get merged.
.default: &default
services:
- postgresql
script:
- default_step1
- default_step2
test1:
<<: *default
# *default will insert the services: posgresql here.
script:
# here the default steps will not be inserted, because the definition of scripts here, will overwrite the scripts section from &default
- job_test1_step1
- job_test1_step2
test2:
<<: *default
# only here we have a the default steps and the default services
test3:
script:
- job_test3_step
- job_test2_step2
Proposal
there is a simple solution for this feature request. yaml itself has nearly everything what is needed.
in the current implementation of the gitlab-ci.yml
, we don't need multidimensional arrays. the yaml parser itself is able to parse the following code:
.scripts_one: &scripts1
- extra1
- extra2
.scripts_two: &scripts2
- extra3
test:
stage: test
environment:
name: staging
script:
- step1
- *scripts1
- step2
- *scripts2
this results in (https://yaml-online-parser.appspot.com/):
"test": {
"environment": {
"name": "staging"
},
"stage": "test",
"script": [
"step1",
[
"extra1",
"extra2"
],
"step2",
[
"extra3"
]
]
},
".scripts_two": [
"extra3"
],
".scripts_one": [
"extra1",
"extra2"
]
}
so the only problem we have is, that the script array has to be flattend from the ci before running the scripts.
having the script
and the services
flattend (and all other parameter which can be an array), would solve the problem. after this change the yaml above would be the same as:
test:
stage: test
environment:
name: staging
script:
- step1
- extra1
- extra2
- step2
- extra3
the combination with the already working merge of hashes, would allow this (n.b. &default
and &default_services
):
.default: &default
service: &default_services
- mysql
- elasticsearch
script:
- env
- ls -lsa
- pwd
test:
<<: *default
stage: test
environment:
name: staging
service:
- *default_services
- postgresql
which is resulting in (with nested arrays):
{
"test": {
"environment": {
"name": "staging"
},
"script": [
"env",
"ls -lsa",
"pwd"
],
"service": [
[
"mysql",
"elasticsearch"
],
"postgresql"
],
"stage": "test"
},
".default": {
"service": [
"mysql",
"elasticsearch"
],
"script": [
"env",
"ls -lsa",
"pwd"
]
}
}
also here the flatten array approach would solve the problem of not having easy job/script templates.
flattening itself looks as nobrainer to me:
s = [ 1, 2, 3 ] #=> [1, 2, 3]
t = [ 4, 5, 6, [7, 8] ] #=> [4, 5, 6, [7, 8]]
a = [ s, t, 9, 10 ] #=> [[1, 2, 3], [4, 5, 6, [7, 8]], 9, 10]
a.flatten #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
https://ruby-doc.org/core-2.2.2/Array.html#method-i-flatten https://stackoverflow.com/a/30770740/590247
it would also be no BC break because the CI currently raises an error on multidimensional arrays, instead of the error -> do the flattening.
Documentation blurb
share code snippets between jobs/stages
:
.default: &default
service: &default_services
- mysql
- elasticsearch
script:
- env
- ls -lsa
- pwd
test:
<<: *default
stage: test
environment:
name: staging
service:
- *default_services
- postgresql
Feature checklist
Make sure these are completed before closing the issue, with a link to the relevant commit.
-
Feature assurance -
Documentation -
Added to features.yml