Investigation Backend: Global variables must be explicitly forwarded to nested child pipeline, which overrides job variables
Summary
Documentation states that global variables are passed to child pipelines implicitly:
- https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#pass-yaml-defined-cicd-variables states that by defining global variables in the parent CI config, they are automatically passed to a child pipeline, where they become 'trigger variables': "all jobs in a pipeline, including trigger jobs, inherit global variables"
- https://docs.gitlab.com/ee/ci/variables/index.html#cicd-variable-precedence lists 'trigger variables' at the very top of the variable precedence chain - above even global variables.
- https://docs.gitlab.com/ee/ci/triggers/index.html#pass-cicd-variables-in-the-api-call implies that 'trigger variables' have the highest priority of all variables and will always be passed to everything: "These variables have the highest precedence, and override all variables with the same name."
Therefore, when passing variables from a parent to a child pipeline, I expect those variables to also be passed to children of that child pipeline.
However, when starting a nested child pipeline, global variables defined in the parent are not passed to it.
There is a workaround: unless the nested child pipeline is triggered with forward: pipeline_variables: true
- which documentation states is reserved for "manual pipeline variables and scheduled pipeline variables" - not global or trigger variables. (https://docs.gitlab.com/ee/ci/yaml/index.html#triggerforward)
However, there is an issue with that workaround. Job-level variables are expected to take precedence over global variables. When the intermediate child defines job-level variables but also explicitly passes global variables with forward: pipeline_variables
, the forwarded global/trigger variables erroneously take precedence over the job-level variables. There is no workaround for this - as far as I know there is no way to override the global variable once forwarded.
Steps to reproduce
Create 5 jobs in 3 pipelines.
-
job1
definesGLOBAL_VAR
and creates a pipeline containingjob2
,job2-forward
andjob2-forward-new-value
, which both create pipelines containingjob3
.-
job2
callsjob3
without explicitly forwarding any variables. -
job2-forward
callsjob3
and explicitly forwardspipeline_variables
. -
job2-forward-new-value
callsjob3
, reassignsGLOBAL_VAR
with a job variable intended to override it, and explicitly forwardspipeline_variables
.
-
-
job3
reports the value ofGLOBAL_VAR
.- In
job2 -> job3
,GLOBAL_VAR
is not set.❌ Unexpected result. - In
job2-forward -> job3
,GLOBAL_VAR
is set.✅ Workaround. - In
job2-forward-new-value -> job3
,GLOBAL_VAR
is set to the original value, not to the override.❌ Unexpected result with no workaround.
- In
# .gitlab-ci.yml
variables:
GLOBAL_VAR: 'true'
job1:
trigger:
include: child.gitlab-ci.yml
# child.gitlab-ci.yml
# This triggered pipeline will fail
job2:
trigger:
include: nested-child.gitlab-ci.yml
# This triggered pipeline will pass
job2-forward:
trigger:
include: nested-child.gitlab-ci.yml
forward:
pipeline_variables: true
# This triggered pipeline will pass, but is expected to fail
job2-forward-new-value:
variables:
GLOBAL_VAR: 'false'
trigger:
include: nested-child.gitlab-ci.yml
forward:
pipeline_variables: true
# nested-child.gitlab-ci.yml
job3:
script:
- echo "The value of GLOBAL_VAR is $GLOBAL_VAR"
# Fail the job if the global var is not set
- test "$GLOBAL_VAR" = "true"
Example Project
https://gitlab.com/rossjrw/nested-child-pipeline-test
See pipeline run: https://gitlab.com/rossjrw/nested-child-pipeline-test/-/pipelines/829069022
Note that job2 -> job3
failed, job2-forward -> job3
passed, and job2-forward-new-value -> job3
passed (but should have failed).
What is the current bug behavior?
- Global variables are not recursively passed to child pipelines. Explicitly forwarding
pipeline_variables
is required. - Job variables do not override global variables when
pipeline_variables
are forwarded.
What is the expected correct behavior?
For bug behaviour 1:
Global variables should be recursively passed to child pipelines.
OR
Documentation should be clear in that a) a parent's global variables become a child's trigger variables, b) a parent's trigger variables are considered manual variables for the purposes of variable forwarding.
For bug behaviour 2: Job-level variables should be passed to downstream pipelines with a higher priority than trigger variables passed via pipeline_variables
forwarding.
Output of checks
This bug happens on GitLab.com