I was expecting a deploy to run on a runner with a tag name the same as the name of my branch (to have the proper environment's deployment variables securely injected).
Instead, I got a hung build where it was looking for a runner with the tag $CI_BUILD_REF_NAME.
Proposal
When evaluating a .gitlab-ci.yml file to schedule a job, first perform variable substitution prior to the logic used for selecting a runner.
Additionally consider implementing regex-style variable manipulation as proposed in #24044 (closed). These two improvements combined would be especially powerful.
Note that $CI_BUILD_REF_NAME is a variable injected by the gitlab CI process, and my concern is how it gets used by the component of gitlab which decides where a job is scheduled. #18758 (moved) appears to be about what happens at the runner level with a scheduled job. It's in fact already possible to use variables in execution scripts and I'm unclear exactly what the user in #18758 (moved) is trying to achieve that can't already be done.
4
Mark FletcherAdded ~19173 ~14213 and removed ~65372 labels
I'd also like to see support for job tags variable expansion.
In our use case, each person on the project has their own separate runner. I was hoping to use the $GITLAB_USER_EMAIL variable in a job tag to know which committer's runner to use based on the user email. I think there's other ways around this with some conditional scripts but I did find this proposal interesting/useful.
Yes we are looking for a similar feature , where we can do variable substitution (not only limited to inbuilt gitlab variables but user defined variables) for TAGS & ENVIRONMENT section of gitlab CI. It will be a really handy feature & you can create a dynamic pipeline using this feature.
@MrChrisW we actually don't support variable expansion for tags, as it is used to check if a job must be scheduled or not. Having variables considered at that point probably means that we've to change some logic that is now on the runner. Maybe a possible workaround is to evaluate during the job run using a script and access the variable at that point.
We could evaluate this in the future, so please feel free to push it if you think it is valuable.
@ayufan could you please elaborate how this request could impact the current implementation of the logic? We could try to see if once we'll have #20367 (closed) maybe an MVP here is possible to achieve with small effort.
@bikebilly My problem here is that we need to sit to the drawing table and group variables into ones that could be used and not.
Some of the variables that we have, are dynamic (job id), some of them are static (job name). If we would have to evaluate dynamic ones in that context it creates a lot of potential problems.
If we figure for what uses cases we could use only: with variables it would allow us to better understand if this is something that we could do.
@ayufan thanks I see the problem now. Dynamic variables are very complex to evaluate and I agree it is probably feasible just for the static ones. This could be enough for this scenario and similar ones, and better than the actual situation. It must be really clear which are usable and which are not, documenting them and maybe thinking about some smart way to check and notify, but it could be a further improvement.
In case we decide to proceed with just static variables supported, which could be the engineering effort?
It is hard to estimate, it depends on what variables would have to be supported. Fortunately, it would allow us to clean up port of configuration processing, so I expect it to be Medium-sized task.
@ayufan cool, I put the %"Next 2-3 months" so we can consider having a close look to it in the near future and see if we can have a simple first iteration that has a value!
Hello,
I have several runners running on different kubernetes clusters, I would like to be able to use variables substitutions to run the same gitlab-ci yaml file on the different clusters based on the variables I set through the API.
Is there anyway to do a workaround for this? I tried using yaml anchors but the result is the same.
Thank you
Hello, we have serveral runners in a single kubernetes cluster. The difference between them is the amount of cpus and memory they can consume. I want to use environment variables from the secret variables page instead of changing the .gitlab-ci.yml file to specify a runner. Thank you
@jlenny , will #18758 (moved) cover also tags - what is specifically requested in this Issue?
There is quite a lot of discussion here with use cases etc, will all this knowledge be lost if Issue is closed?
Whoever works on #18758 (moved) will they look at this one too even if it's in Closed state?
Would love this too, we have several runners, but only 1 particular runner has a certain build requirement.
90% of the time our builds are fine cause it works on all runners, but we then need to compile it for an older JDK that is only available on that particular runner.
Currently have to always run it on that runner even for the normal use case and it gets hammered.
Hello, any chances that it's going to be implemented?
I came up with brilliant idea to run my jobs on cloud container services (dynamically create it and delete afterwards), but without specifying tag as variable I'm unable to point my job on right runner...
This is an interesting feature, but not one that we currently yet have scheduled to a specific release. If you're interested in contributing, we accept contribution on all items in our backlog.
Elliot Rushtonadded 1 deleted label and removed 1 deleted label
@cojack You are more than welcomed to do it yourself. It is an open core project. It would be more helpful than patronizing fellow software engineers. Although, I do agree that this would be a very useful feature.
It is indeed an open-source project but there're EE customers that are paying good money for the product and GL has engineers on the payroll that can spend a couple of hours implementing what customers demand. Being an OSS project maintainer myself I also get frustrated by comments like "why is it not done yet" but I clearly make a distinction between community-driven and vendor-driven OSS projects.
I ended up here from a google search looking for a way to use variables in tags. I merge into different branches to move the ci/cd pipeline into different environments. I would like to tag a runner with the branch name and be able to use the CI_COMMIT_REF_NAME to select the runner.
@mitar: A couple of weeks ago we moved all development into a single codebase as outlined in this blog post. With https://gitlab.com/gitlab-org/gitlab/ now being the main development repository and this feature being requested by multiple users and enterprise customers, it would give the issue more visibility to our internal teams if this issue were re-opened in that project. This project, https://gitlab.com/gitlab-org/gitlab-foss is the read only mirror to the other repository. I moved all relevant discussion and internal/external links to that issue, so if this is important to you, please leave feedback at gitlab#35742 (closed). Apologies for the inconvenience!