!reference creates nested arrays, making it unuseful for many purposes
For instance see the sample code in the documentation: https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags
.snippets:
one:
- echo "ONE!"
two:
- !reference [.snippets, one]
- echo "TWO!"
three:
- !reference [.snippets, two]
- echo "THREE!"
nested-references:
script:
- !reference [.snippets, three]
Based on the documentation I would have expected this to expand to something like this:
".snippets":
one:
- echo "ONE!"
two:
- echo "ONE!"
- echo "TWO!"
three:
- echo "ONE!"
- echo "TWO!"
- echo "THREE!"
"nested-references":
script:
- echo "ONE!"
- echo "TWO!"
- echo "THREE!"
But actually, if you run this through the pipeline editor and click "view merged YAML" you'll see the following output (intendentation original):
".snippets":
one:
- echo "ONE!"
two:
- - echo "ONE!"
- echo "TWO!"
three:
- - - echo "ONE!"
- echo "TWO!"
- echo "THREE!"
"nested-references":
script:
- - - - echo "ONE!"
- echo "TWO!"
- echo "THREE!"
The documentation implies that this is functionality is suitable for list concatenation, which is sorely needed (e.g. for lists of rules, or changes conditions, or whatever), but the above proves it is not. Instead it does a very weird nested list structure of some kind, which seems to work mostly accidentally as shell scripts, but is unuseful for anything else.
It's hard for me to believe this isn't a bug, but if this behavior is intentional, it would be useful to document it clearly on that page, since this appears to solve an important problem, but mostly doesn't.
--
If I can guess at the cause, it looks like when you write - !reference[whatever]
then it prepends the value of that reference with a -
. So the obvious "fix" for the user is to leave the -
off, which "should" fix the list nesting issue. Unfortunately the gitlab yaml parser rejects that as invalid yaml.
You can write something like this whatever: !reference [whatever]
which doesn't cause the nesting, but also isn't suitable for combining references with each other or with explicit code.
A couple possible fixes spring to mind:
- Stop the nesting, since it seems to be a bug (I'm surprised the generated yaml is even valid, honestly)
- Allow the use of !reference in a list without putting your own
-
in front of it (this would mean the manual yaml isn't yaml anymore, which is undesirable) - Make a new custom tag, like !reference, which flattens (maybe !flat-reference or something); this would preserve back compatibility and so on. I think there is a lot of interest in flattening / concatenating lists anyway so maybe a !flatten tag. I don't know. Something like that.