Backend: Variable expansion in dynamic child pipeline with nested includes

Summary

Variable expansion does not happen in a dynamic child pipeline with nested includes.

Steps to reproduce

  1. Create a group level CICD variable named JFROG_ARTIFACTORY_SELECTOR and set its value to "test" (group used for this test is named superkas)
  2. Create a project and add a file that will be used to trigger dynamic child pipeline.

Project: superkas/includes-test

File: artifacts/functionbeat_rendered.ci.yml

File content:

include:
  - project: 'superkas/gitlab-ci-shared'
    ref: main
    file:
      - 'rwwa-common.ci.yml'
  1. Add .gitlab-ci.yml file:
stages:
  - build_pipelines
  - run_pipelines
​
default:
  tags: ["gitlab-gcp-k8s-15-0"]
​
.build_pipeline_agent_deploy:
  stage: build_pipelines
  variables:
    GIT_STRATEGY: clone
    GIT_DEPTH: 1
  script:
    # Get the templater Scripts & jinja file
    - echo "$CI_JOB_NAME"
    - cp artifacts/functionbeat_rendered.ci.yml .
  artifacts:
    paths:
      - '*_rendered.ci.yml'
​
functionbeat_deploy_build:
  extends: .build_pipeline_agent_deploy
  variables:
    ELASTIC_AGENT_NAME: "functionbeat"
​
functionbeat_child:
  stage: run_pipelines
  trigger:
    strategy: depend
    include:
      - artifact: functionbeat_rendered.ci.yml
        job: functionbeat_deploy_build
  variables:
    ELASTIC_AGENT_NAME: "functionbeat"
  needs:
    - job: functionbeat_deploy_build
      artifacts: true
  1. Create the project and the yaml file (referenced in the include keyword in functionbeat_rendered.ci.yml):

Project: superkas/gitlab-ci-shared

File: rwwa-common.ci.yml

File content:

include:
  - project: 'superkas/gitlab-ci-shared'
    ref: main
    file:
      ### JF CLI Support ###
      - artifactory-selectors/$JFROG_ARTIFACTORY_SELECTOR.ci.yml
  1. Create artifactory-selectors/test.ci.yml in superkas/gitlab-ci-shared project:
deploy:
  stage: deploy
  script: 
    - echo "$CI_JOB_NAME"
  1. Run the pipeline. You will get an error. This is because JFROG_ARTIFACTORY_SELECTOR variable hasn't been expanded.

Found errors in your .gitlab-ci.yml:

• Project 'superkas/gitlab-ci-shared' file "artifactory-selectors/.ci.yml' does not exist!

Screen_Shot_2022-11-15_at_8.00.18_PM

Example Project

https://gitlab.com/gitlab-gold/emunn-test/kas/nested-includes/-/pipelines/702524602

What is the current bug behavior?

Group level CICD variable, JFROG_ARTIFACTORY_SELECTOR is not available in the nested include file. This makes the following include to fail because file artifactory-selectors/.ci.yml cannot be found:

include:
  - project: 'superkas/gitlab-ci-shared'
    ref: main
    file:
      ### JF CLI Support ###
      - artifactory-selectors/$JFROG_ARTIFACTORY_SELECTOR.ci.yml

What is the expected correct behavior?

artifactory-selectors/$JFROG_ARTIFACTORY_SELECTOR.ci.yml should resolve to artifactory-selectors/test.ci.yml.

Relevant logs and/or screenshots

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by Furkan Ayhan