Self-cyclic CI/CD variables not handeled correctly

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

If a variable appears on the left and right side of a variable assignment in .gitlab-ci.yml, it is not identified as cyclic dependency which leads to unexpected behavior.

  variables:
    MY_VAR_A: "self reference is [${MY_VAR_A}]"

Steps to reproduce

Create a CI/CD job with a variable self-reference or single-item cycle in the variable substitution DAG.

test:
  variables:
    MY_VAR_A: "self reference is [${MY_VAR_A}]"
  script:
    - env | grep "MY_" | sort
    # Produces
    # MY_VAR_A=self reference is [self reference is [${MY_VAR_A}]]

Example Project

https://gitlab.com/sauerburger/single-variable-cycle

CI output: https://gitlab.com/sauerburger/single-variable-cycle/-/jobs/10748808413

What is the current bug behavior?

Variable substitution to a variable itself is not identified as a cyclic dependency during DAG / pipeline validation.

MY_VAR_A=self reference is [self reference is [${MY_VAR_A}]]

After the topological sort, a single variable substitution is applied where the inner variable is kept literally.

In more simple cases, where not additional strings are prefixed or appended, e.g., APPKEY="$APP_KEY", it appears to the user as if no variable substitution has happened at all.

What is the expected correct behavior?

A CI variable referencing itself should be identified as a cyclic dependency and make the pipeline fail.

Relevant logs and/or screenshots

Using effective pull policy of [always] for container ruby:3.1
Using docker image sha256:9981df1d883b246c27c62f8ccb9b57d3e07d14cee8092299e102b4a69c35ea61 for ruby:3.1 with digest ruby@sha256:91627f55e8969006aab67d15c92fb930500ff73948803da1330b8a853fecebb5 ...
$ env | grep "MY_"
MY_VAR_A=self reference is [self reference is [${MY_VAR_A}]]
Cleaning up project directory and file based variables 00:00
Job succeeded

Output of checks

This bug happens on GitLab.com

Possible fixes

  1. During pipeline validation, the variable DAG is checked for cyclic dependencies (https://gitlab.com/gitlab-org/gitlab/-/blob/78b109bbe363cfa5e28425b8239e29beed57a804/lib/gitlab/ci/yaml_processor/dag.rb#L18).
  2. This is done using the topological sort TSort. sort raises an exception if cyclic dependencies are identified.
  3. Cyclic dependencies are identified if a strongly connected component has size larger than 1 (https://github.com/ruby/ruby/blob/d21e4e76c44b3be940c4fd8be6a649cdf366f0f9/lib/tsort.rb#L232)
  4. A variable referencing itself is a strongly connected component of size 1, and therefore doesn't trigger an exception.

The problem can be fixed by adding an explicit check that no variable references itself. I don't think the topological sort implementation can help.

I'm more than happy to contribute to a merge request.

Complication

The problem becomes more subtle as CI/CD variables follow a hierarchy. A user might be tempted to use a global variable in a local variable of the same name.

variables:
  MY_VAR_A: "global value"

test:
  variables:
    MY_VAR_A: "self reference shadowing global variable is [${MY_VAR_A}]"

The local variable shadows the global variable. The example does not lead to "self reference shadowing global variable is [global value]".

Instead the local variable is a self-reference and give the exact same result as above: MY_VAR_A=self reference is [self reference is [${MY_VAR_A}]]

The correct behavior should be again a failing pipeline.

Relevancy

The same variable on the left and right is a very common practice in tools like docker-compose, where a variable in the CLI environment is mapped to the same variable inside the container.

service:
  app:
     ...
     environment:
        APP_KEY: "$APP_KEY"
Edited by 🤖 GitLab Bot 🤖