Backend: Pre-filled variables do not work correctly for branches for external custom CI config file paths

Summary

#336184 (closed) solved the problem of loading pre-filled variables for projects using custom CI config file paths. However, the solution does not handle branches correctly: when running a pipeline manually selecting non-default branch results in no variable being picked up. Pre-filled variables are loaded from external project using selected branch instead of using default branch name

Explaination from customer:

This was initially reported by Customer. This is their explanation:

We have a problem, when we try using Predefined global variables in custom CI/CD configuration file. We have special project, which contain only .gitlab-ci.yml files for other projects. For example: We have .gitlab-ci.yml file stored in project group1/project1 (in this project we have only one branch, with name "master"). In this file we have strings:

variables:
  GIT_STRATEGY: none
 
  DEPLOY_TO_TEST:
    value: 0
    description: "Change this variable to '1' if needed deploy to test"

  DEPLOY_TO_STAGE:
    value: 0
    description: "Change this variable to '1' if needed deploy to stage"

  SKIP_TEST:
    value: 0
    description: "Skip maven tests"

In project group1/project2, menu "Settings" - "CI/CD" - "General pipelines" - string "CI/CD configuration file", we specify: .gitlab-ci.yml@group1/project1

After that we try run new pipeline manually, in menu "CI/CD" - "Pipelines" - press button "Run pipeline". If we specify branch master in field "Run for branch name or tag", predefined variables specified in .gitlab-ci.yml file in project group1/project1 displayed in web page. If we specify another branch in field "Run for branch name or tag" (for example branch "develop"), predefined variables not displayed in web page.

We create new branch "develop" in project group1/project1, and after few moments later (we don't know how long it took) we choose "develop" branch, when try run pipeline manually in project group1/project2, and predefined variables display in web page.

I think this is wrong behavior. Maybe we're missing something when using custom CI/CD configuration file?

Steps to reproduce

  1. Create project X
  2. In project X create a .gitlab-ci.yml
  3. In the above file, add pre-defined variables
  4. Try to run manual pipeline on project X
  5. Pre-filled variables are shown in UI as expected
  6. Create project Y
  7. Use settings of project Y to point its CI config path to project X: .gitlab-ci.yml@X
  8. Try to run a manual pipeline on project Y
  9. Pre-filled variables are shown in UI as expected
  10. Create branch develop in Project X
  11. Try to run a manual pipeline on project Y, branch develop
  12. Pre-filled variables are not shown on the UI

Example project

https://gitlab.com/dzalbo-group/sandbox/external-pipeline-config/application (happy to give access on request)

Considered workaround

I have considered a workaround of setting Custom CI/CD configuration file on Project Y to .gitlab-ci.yml@X:main (adding a reference name), but in this case pre-filled variables don't load at all. This also looks like undesirable behaviour as our documentation clearly allows my/path/.my-custom-file.yml@mygroup/another-project:refname format

The only way to load pre-filled variables on non-default branch is to create a branch on Project X with the same name, but that's not a real workaround

What is the current bug behavior?

Pre-filled variables defined in custom CI config path outside of repository does not render when selecting a branch

What is the expected correct behavior?

Pre-filled variables defined in custom CI config path outside of repository render in run pipeline form independently of selected branch

Edited by Vladimir Dzalbo (GitLab)