Skip to content

Preserve relative order of overridden variables

What does this MR do?

This MR fixes an issue found during the last deployment of the :variable_inside_variable FF (it is currently disabled waiting for this MR). The issue has to do with the new topological sorting that happens on variables when the feature flag is enabled. It can be reproduced when there are multiple instances of a variable with the same name due to it being overridden in parent subgroups. Here is a sample of the ordering when the FF is disabled:

[
  "TEST=$AWS_ACCOUNT_NONPROD",
  "GITLAB_USER_ID=1",
  "GITLAB_USER_EMAIL=admin@example.com",
  "GITLAB_USER_LOGIN=root",
  "GITLAB_USER_NAME=Administrator",
  "AWS_ACCOUNT_NONPROD=gitlab-org",
  "AWS_ACCOUNT_NONPROD=vars-in-vars-subgroup"
]

Once we enable the FF, we see AWS_ACCOUNT_NONPROD correctly being moved before the variable that references it (TEST), but since we keep the definitions of AWS_ACCOUNT_NONPROD from multiple group levels, the definition from parent groups end up overwriting the first value and the top-level group value wins:

[
  "AWS_ACCOUNT_NONPROD=vars-in-vars-subgroup",
  "TEST=vars-in-vars-subgroup",
  "GITLAB_USER_ID=1",
  "GITLAB_USER_EMAIL=admin@example.com",
  "GITLAB_USER_LOGIN=root",
  "GITLAB_USER_NAME=Administrator",
  "AWS_ACCOUNT_NONPROD=gitlab-org"
]

The topological sorting algorithm needs to be improved so that it moves all instances (preserving order) of a referenced value before the reference variable, not just the first one:

[
  "AWS_ACCOUNT_NONPROD=gitlab-org",
  "AWS_ACCOUNT_NONPROD=vars-in-vars-subgroup",
  "TEST=vars-in-vars-subgroup",
  "GITLAB_USER_ID=1",
  "GITLAB_USER_EMAIL=admin@example.com",
  "GITLAB_USER_LOGIN=root",
  "GITLAB_USER_NAME=Administrator"
]

Screenshots or Screencasts (strongly suggested)

How to setup and validate locally (strongly suggested)

  1. In a GDK instance, create the following project structure:

    • toplevel-playground-group
      • playground-subgroup
        • project.
  2. In toplevel-playground-group, define AWS_ACCOUNT_NONPROD as top-level value in the group CI Variables page

  3. In toplevel-playground-group / playground-subgroup, define AWS_ACCOUNT_NONPROD as subgroup value in the group CI Variables page

  4. In project, create a .gitlab-ci.yml file with the following content:

    stages:
      - test
    
    variables:
      AWS_ACCOUNT: "${AWS_ACCOUNT_NONPROD}"
    
    test-job:
      stage: test
      script:
        - |
          echo "AWS_ACCOUNT_NONPROD: $AWS_ACCOUNT_NONPROD"
          echo "AWS_ACCOUNT: $AWS_ACCOUNT"
  5. Enable the variable inside variables logic globally in rails console:

    Feature.enable(:variable_inside_variable)
  6. Run a pipeline

  7. Both AWS_ACCOUNT and AWS_ACCOUNT_NONPROD should appear as subgroup value, which is the same expected result as when the FF is disabled.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Part of gitlab-runner#1809 (closed)

Merge request reports