Allow for variable substitution in CI tag names (Dynamic Tags)

added CI variables customer customer+ devopsverify typefeature labels

mentioned in issue gitlab-foss#24207 (closed)

Thanks julianne for raising this feature request

We would really appreciate if this request can be prioritize

I merge into different branches to move the ci/cd pipeline into different environments. I would like to tag a runner with the branch name and be able to use the CI_COMMIT_REF_SLUG to select the runner.

mentioned in issue #35979 (closed)

mentioned in issue #36570 (closed)

added Category:Continuous Integration grouppipeline execution labels

mentioned in issue #37193 (closed)

mentioned in issue #37871 (closed)

Customer https://gitlab.my.salesforce.com/0016100001Xv01k is interested in this feature. @thaoyeager could we set up a call to detail the cust use case and potential workarounds?

@bcupini Yes, happy to get on a call to understand the use case and potential workarounds.

What'd y'all come up with?

A Premium customer would like this feature - ZD (internal use only)

mentioned in issue #39013 (closed)

added auto updated potential proposal labels

mentioned in issue #103415 (closed)

mentioned in issue #121492 (closed)

mentioned in issue #193038 (closed)

mentioned in issue #195111 (closed)

@thaoyeager happy new year, picking up this thread for my customer. I'd like to arrange a call w/ the customer and you / your team next week to detail their use case and identify workarounds (if any), but can we sync up first so I can give you the backstory on the notes from my meeting? I'll reach out on Slack also. Thanks!

@bcupini Hello! I'm guessing since we did not connect on Slack that a call with the customer has not yet been scheduled. When ready, please coordinate with @jyavorska who is helping out as interim PM for Verify:CI.

@bcupini I've marked this as a duplicate of gitlab-runner#1809 (closed) for which @DarrenEastman is the PM, and is the more broad issue that is a single case of. He would be the right PM to chat with.

@bcupini My calendar is up to date so feel free to schedule the call. @jyavorska

mentioned in issue #196475 (closed)

mentioned in issue #197809 (closed)

This appears to be a duplicate of gitlab-runner#1809 (closed) which is seeking a general solution for allowing variable substitution within other variables. Please let me know if I've missed something here, though.

I believe gitlab-runner#1809 (closed) should remain a separate issue or updated to explicitly state that job tags should be made dynamic:

Currently a tag is just a constant string. You cannot use any variables in a tag. But gitlab-runner#1809 (closed) is about using nested variables (A -> B -> C).
Correct me if I'm wrong, but variable substitution in gitlab-runner#1809 (closed) is done after runner instance has already been selected. And dynamic tag should be used before that to select an appropriate runner instance.

@vitaly.ogoltsov We have updated the proposed implementation plan in gitlab-runner#1809 (closed) to handle expansion of known variables in GitLab Rails. However, in reading this issue, I don't think that gitlab-runner#1809 (closed) will address this use case of tags within the gitlab-ci.yml file being able to utilize variables.

@ayufan Can you comment?

@DarrenEastman

It is not possible, and this is different problem to solve in different context.

We do not perform variable substitution in some of the contexes, and tags: is one of them.

So, I don't think that this is duplicate, rather this is a feature request to allow "dynamic tags:".

@jyavorska I propose that we keep this issue open per @ayufan feedback.

Sure thing @DarrenEastman

We also need this variable runner tag feature or "dynamic tags", in order to standardize and use centralized templates for pipelines that run on different servers depending on the project or deployment environment. This way we would be able to handle hundreds of different project repositories that reference and include the same pipeline template stored on a shared repository, running on the proper gitlab runner for each project.

marked this issue as a duplicate of gitlab-runner#1809 (closed)

closed

marked this issue as a duplicate of gitlab-runner#1809 (closed)

marked this issue as related to gitlab-runner#1809 (closed)

reopened

mentioned in issue #198992 (closed)

changed milestone to %Backlog

added [deprecated] Accepting merge requests label

A silver customer would like to have this feature as well https://gitlab.zendesk.com/agent/tickets/146356 (internal use only)

Any new features to utilize the schedule variables to substitute the "tags:" in gitlab-ci.yml file?

Hello, I've got the same problem. Having this feature would avoid me to use multiple gitlab-ci.yml files depending on branches and environnement.

Edit : For the moment and as a temporary solution i did the trick with a YAML template but it's very redundant

.random-template: &random_template
  stage: random
  image: $DummyImage
  script:    
    echo "Hello World"


job:dev:
  <<: *random_template
  tags: 
    - dev
  only:
    - develop

job:prod:
  <<: *random_template
  tags: 
    - prod
  only:
    - master

I'm trying to wrap my head arround this solution, and I don't see how I can use this to select/target specific runner based on tags:
Could you please explain it a bit more?

I'm creating on-demand runners on google cloud. I want to achieve that single runner runs single pipeline, so If i have two pipelines in one project, I want to have two runners running each of the pipelines.

Does your solution applies to this kind of scenario?

Thank you in advance :)

mentioned in issue #15356 (closed)

marked this issue as related to #37267 (closed)

since #15356 (closed) is scheduled for 13.0 and would be much, much more useful if parallel/matrix jobs could be assigned to different runners via variables, could we get this scheduled for 13.0 or 13.1 as well?

Another Silver tier customer requesting this feature: https://gitlab.my.salesforce.com/0016100001RE5Ol

A Premium customer is also interested in this feature: https://gitlab.zendesk.com/agent/tickets/155782

would be of GREAT help if we can use variables in tags as well this seems to be a common use case.

mentioned in issue gitlab-runner#1809 (closed)

changed title from Allow for variable substitution in CI tag names to Allow for variable substitution in CI tag names (Dynamic Tags)

I agree that having dynamic tags as suggested above would allow for more standardized CI templates to be created and used across many different projects. For example, the following could be a desired state (using an example configuration entry that would be inherited from):

.tags:
  tags:
    - ${SITE}-ansible
  environment: "$SITE"

Assuming here I have $SITE defined to a specific environment, and I have a $SITE-ansible runner designated for that environment. My goal here would be to let a tag for a runner be chosen based on the variable I pass to the pipeline. The reason this is extremely valuable is it allows a single project or CI file to define pipelines for multiple sites or environments while minimizing repeated code.

Current workarounds have been to either create multiple projects to logically separate deployments or create duplicate job definitions with different hardcoded tags. This leaves quite a bit of CI code behind to maintain and update as new features are developed and code is refactored.

Recently, I leveraged the Dynamic child pipelines feature to achieve the same goal. While this involves spawning a child pipeline instead of having the single "base" pipeline choose dynamic tags, this solution does have the same functionality.

In the base .gitlab-ci.yml file (example below), the first job utilizes a Python script to generate a complete dynamic gitlab-ci file from a template (also below) which is then exported as a job artifact. The Python script leverages the jinja2 library to fill the template out with the appropriate variable(s). This artifact is then passed to the next job which leverages the "dynamic" child pipeline feature to create the desired pipeline.

.gitlab-ci.yml

---
stages:
  - setup
  - triggers

generate_config:
  stage: setup
  variables:
    SITE: svl-fab3-lab3
  script:
    - echo $SITE
    - pip install pyyaml jinja2
    - python generate_ci_template.py
  artifacts:
    paths:
      - .gitlab-ci-complete.yml

trigger_site:
  stage: triggers
  trigger:
    include:
      - artifact: .gitlab-ci-complete.yml
        job: generate_config
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'

Relevant subsection of .gitlab-ci-template.yml.j2 below. After generate_ci_template.py fills this Jinja template out, it is saved as .gitlab-ci-complete.yml and set as an artifact.

.tags:
  tags:
    - "{{ gitlab_ci_tag }}"
  environment: "$SITE"

The documentation here was particularly useful: https://docs.gitlab.com/ee/ci/parent_child_pipelines.html#dynamic-child-pipelines

I know this doesn't solve the need for dynamic tags, but I was able to use it to gain the same level of abstraction while being able to transition to dynamic tags when available.

mentioned in issue #220875 (closed)

Allow for variable substitution in CI tag names (Dynamic Tags)

Note

Problem to solve

Intended users

Further details

Proposal

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references

Designs

Child items 0

Activity

Allow for variable substitution in CI tag names (Dynamic Tags)

Note

Problem to solve

Intended users

Further details

Proposal

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references

Is blocked by

Relates to

Activity