Skip to content

Fix file variable expansion in downstream pipelines

Albert requested to merge 390252-forward-file-variable-implicitly into master

Part 3 of #390252 (closed)

What does this MR do and why?

  1. Prevent file variable expansion content in downstream pipeline variables.
  2. Implicitly forwards file variable dependencies as file type to downstream pipelines when they are interpolated in a variable. This will ensure that the downstream pipeline has the file variable in the file system.

MR dependencies:

  1. !121215 (merged) - adds a keyword argument to ExpandVariables to choose between expanding file variables or not.
  2. !124580 (merged) - refactors downstream variables in Ci::Bridge.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After
Child pipeline Screenshot_2023-06-27_at_3.44.47_PM Screenshot_2023-06-27_at_3.44.23_PM
Multi-project pipeline Screenshot_2023-06-27_at_3.47.18_PM Note that the file variable is created in the downstream project path: Screenshot_2023-06-27_at_3.47.28_PM

How to set up and validate locally

  1. Create a group test-group.
  2. Create 2 projects test-group/main-project and test-group/downstream-project
  3. In test-group, create a File variable GROUP_FILE_VARIABLE.
  4. In test-group/main-project, create a File variable PROJECT_FILE_VARIABLE.
  5. In test-group/main-project, create the following CI YMLs for the parent pipeline and child pipeline.
  6. In test-group/downstream-project, create the following CI yml
  7. Run pipeline manually on test-group/main-project, specifying a File variabel PIPELINE_FILE_VARIABLE.
  8. Observe the results of the echo jobs
  9. Enable feature flag :ci_prevent_file_var_expansion_downstream_pipeline
  10. Repeat steps to run pipeline manually
  11. Observe the results of the echo jobs

main-project CI YMLs

# .gitlab-ci.yml
variables:
  GROUP_FILE_VARIABLE: "$GROUP_FILE_VARIABLE"
  PROJECT_FILE_VARIABLE: "$PROJECT_FILE_VARIABLE"
  PIPELINE_FILE_VARIABLE: "$PIPELINE_FILE_VARIABLE"

echo:
  script:
    - echo "GROUP_FILE_VARIABLE $GROUP_FILE_VARIABLE"
    - echo "PROJECT_FILE_VARIABLE $PROJECT_FILE_VARIABLE"
    - echo "PIPELINE_FILE_VARIABLE $PIPELINE_FILE_VARIABLE"

trigger_child:
  trigger:
    include:
      - local: child-pipeline.yml

trigger_downstream_project:
  trigger:
    project: test-group/downstream-project
# child-pipeline.yml
echo:
  script:
    - echo "GROUP_FILE_VARIABLE $GROUP_FILE_VARIABLE"
    - echo "PROJECT_FILE_VARIABLE $PROJECT_FILE_VARIABLE"
    - echo "PIPELINE_FILE_VARIABLE $PIPELINE_FILE_VARIABLE"

downstream-project CI yml

echo:
  script:
    - echo "GROUP_FILE_VARIABLE $GROUP_FILE_VARIABLE"
    - echo "PROJECT_FILE_VARIABLE $PROJECT_FILE_VARIABLE"
    - echo "PIPELINE_FILE_VARIABLE $PIPELINE_FILE_VARIABLE"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #390252 (closed)

Edited by Albert

Merge request reports