The support for registration tokens in the command to register a runner, gitlab-runner register is deprecated. It has begun accepting Runner authentication token in %16.0 in place of registration tokens. The plan is to enable the enforce_create_runner_workflow feature flag by default in GitLab %17.0 so that registration tokens cannot be used to create runners. The enforce_create_runner_workflow feature flag is planned for removal in %18.0, together with support for registration tokens.
NOTE: The following notice displayed by GitLab Runner 15.6 when registering a runner is out of date:
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see #380872 (closed)
The register command will be preserved with some changes, which should limit the impact to users.
High-level overview of changes
As a result of the Next GitLab Runner Token Architecture effort, we are in the process of deprecation runner registration tokens and favoring an alternative process. The new process consists of (1) creating a runner directly in the GitLab UI, (2) getting an
authentication token in return, and (3) using that authentication token in place of the registration
token.
This has added benefits including preserving ownership records for runners, while minimizing the
impact on users.
Reusing the same authentication token across multiple runners (commonly in an auto-scaling scenario
where a runner manager spawns a runner process with a fixed authentication token) is supported
through the addition of a unique system ID. This ID is generated once at the runner's startup,
persisted in a sidecar file, and sent to the GitLab instance when requesting jobs.
This allows the GitLab instance to display which system executed a given job.
The new registration process is expected to become available in %16.0, and the legacy registration process will be available side-by-side for a few milestones before the being sunset through a feature flag. Removal is planned for %18.0.
graph TD subgraph new[<b>New registration flow</b>] A[<b>GitLab</b>: User creates a runner in GitLab UI and adds the runner configuration] -->|<b>GitLab</b>: creates ci_runners record and returns<br/>new 'glrt-' prefixed authentication token| B B(<b>Runner</b>: User runs 'gitlab-runner register' command with</br>authentication token to register new runner machine with<br/>the GitLab instance) --> C{<b>Runner</b>: Does a .runner_system_id file exist in<br/>the gitlab-runner configuration directory?} C -->|Yes| D[<b>Runner</b>: Reads existing system ID] --> F C -->|No| E[<b>Runner</b>: Generates and persists unique system ID] --> F F[<b>Runner</b>: Issues 'POST /runner/verify' request<br/>to verify authentication token validity] --> G{<b>GitLab</b>: Is the authentication token valid?} G -->|Yes| H[<b>GitLab</b>: Creates ci_runner_machine database record if missing] --> J[<b>Runner</b>: Store authentication token in .config.toml] G -->|No| I(<b>GitLab</b>: Returns '403 Forbidden' error) --> K(gitlab-runner register command fails) J --> Z(Runner and runner machine are ready for use) end subgraph current[<b>Current registration flow</b>] A'[<b>GitLab</b>: User retrieves runner registration token in GitLab UI] --> B' B'[<b>Runner</b>: User runs 'gitlab-runner register' command<br/>with registration token to register new runner] -->|<b>Runner</b>: Issues 'POST /runner request' to create<br/>new runner and obtain authentication token| C'{<b>GitLab</b>: Is the registration token valid?} C' -->|Yes| D'[<b>GitLab</b>: Create ci_runners database record] --> F' C' -->|No| E'(<b>GitLab</b>: Return '403 Forbidden' error) --> K'(gitlab-runner register command fails) F'[<b>Runner</b>: Store authentication token<br/>from response in .config.toml] --> Z'(Runner is ready for use) end style new fill:#f2ffe6
Breaking Change
Yes
Affected Topology
N/A - This change is specific to GitLab Runner.
Affected Tier
Free
Premium
Ultimate
Checklists
Labels
This issue is labeled deprecation, and with the relevant ~devops::, ~group::, and ~Category: labels.
As soon as possible, but no later than the third milestone preceding the major release (for example, given the following release schedule: 14.8, 14.9, 14.10, 15.0 – 14.8 is the third milestone preceding the major release):
Documentation has been updated to mark the feature as deprecated.
On or before the major milestone: A removal entry has been created so the removal will appear on the removals by milestones page and be announced in the release post.
Your stage's stable counterparts have been @mentioned on this issue. For example, Customer Support, Customer Success (Technical Account Manager), Product Marketing Manager.
To see who the stable counterparts are for a product team visit product categories
If there is no stable counterpart listed for Sales/CS please mention @timtams
If there is no stable counterpart listed for Support please mention @gitlab-com/support/managers
If there is no stable counterpart listed for Marketing please mention @cfoster3
Your GPM has been @mentioned so that they are aware of planned deprecations. The goal is to have reviews happen at least two releases before the final removal of the feature or introduction of a breaking change.
Darren Eastmanmarked the checklist item This issue is labeled deprecation, and with the relevant ~devops::, ~group::, and ~Category: labels. as completed
marked the checklist item This issue is labeled deprecation, and with the relevant ~devops::, ~group::, and ~Category: labels. as completed
If you do not feel the purpose of this issue matches one of the types, you may apply the typeignore label to exclude it from type tracking metrics and future prompts.
@atanayno@nelbacha Pinging you for awareness regarding this deprecation notice. The plan is to remove support for the gitlab-runner register command in GitLab 16.0.
Thanks for the ping @DarrenEastman! The linked MRs !102579 (diffs) and gitlab-runner!3702 (diffs) mention deprecation, but they do not seem to describe how it will work after the deprecation (gitlab-runner deploy command is not mentioned there). Should we also add it there?
Hi @DarrenEastman and @pedropombeiro , I see we say here in the MR that we plan to introduce a new GitLab Runner token architecture--can we say when we plan to do that?
The planned milestone for disabling support for registration tokens has been moved to %16.6 to allow users to have sufficient time to experiment with the new workflow.
Pedro Pombeiro (OOO from Feb 17th-21st)marked the checklist item Your stage's stable counterparts have been @mentioned on this issue. For example, Customer Support, Customer Success (Technical Account Manager), Product Marketing Manager. as completed
marked the checklist item Your stage's stable counterparts have been @mentioned on this issue. For example, Customer Support, Customer Success (Technical Account Manager), Product Marketing Manager. as completed
Pedro Pombeiro (OOO from Feb 17th-21st)marked the checklist item Your GPM has been @mentioned so that they are aware of planned deprecations. The goal is to have reviews happen at least two releases before the final removal of the feature or introduction of a breaking change. as completed
marked the checklist item Your GPM has been @mentioned so that they are aware of planned deprecations. The goal is to have reviews happen at least two releases before the final removal of the feature or introduction of a breaking change. as completed
Pedro Pombeiro (OOO from Feb 17th-21st)changed title from Deprecation - gitlab-runner register command to Deprecation - Support for registration tokens in gitlab-runner register command
changed title from Deprecation - gitlab-runner register command to Deprecation - Support for registration tokens in gitlab-runner register command
Pedro Pombeiro (OOO from Feb 17th-21st)changed title from Deprecation - Support for registration tokens in gitlab-runner register command to Deprecation - Support for registration tokens and server-side runner configuration parameters in gitlab-runner register command
changed title from Deprecation - Support for registration tokens in gitlab-runner register command to Deprecation - Support for registration tokens and server-side runner configuration parameters in gitlab-runner register command
Currently, v15.6.1 gitlab-runner register complains about deprecation in favor of deploy. However, as the --help does not list deploy command at all. The users are left wondering and resorting to google (and this issue...) in hopes of finding out how does deploy differ from register and what to expect when issuing gitlab-runner deploy.
Currently, using v15.8.0 and still no mention about the deploy command.
$ gitlab-runner --helpNAME: gitlab-runner - a GitLab RunnerUSAGE: gitlab-runner [global options] command [command options] [arguments...]VERSION: 15.8.0 (12335144)AUTHOR: GitLab Inc. <support@gitlab.com>COMMANDS: exec execute a build locally list List all configured runners run run multi runner service register register a new runner reset-token reset a runner's token install install service uninstall uninstall service start start service stop stop service restart restart service status get status of a service run-single start single runner unregister unregister specific runner verify verify all registered runners artifacts-downloader download and extract build artifacts (internal) artifacts-uploader create and upload build artifacts (internal) cache-archiver create and upload cache artifacts (internal) cache-extractor download and extract cache artifacts (internal) cache-init changed permissions for cache paths (internal) health-check check health for a specific address read-logs reads job logs from a file, used by kubernetes executor (internal) help, h Shows a list of commands or help for one commandGLOBAL OPTIONS: --cpuprofile value write cpu profile to file [$CPU_PROFILE] --debug debug mode [$RUNNER_DEBUG] --log-format value Choose log format (options: runner, text, json) [$LOG_FORMAT] --log-level value, -l value Log level (options: debug, info, warn, error, fatal, panic) [$LOG_LEVEL] --help, -h show help --version, -v print the version
@guilherme.kyte please see the thread directly below for an explanation. If you're on 15. 8, you shouldn't be seeing a deprecation warning mentioning the deploy command, but rather one about support for registration tokens (as mentioned in the title of this issue). Sorry for the change in plans, but this is a complex change and there are many moving parts.
... Well --- it seems that deploy is not there in 15.6.1 at all yet. So I guess the intent is just to inform that register will go out of fashion sometime in the future. For now however, register is still the only way to go. Deprecated, but the only available option.
@mikko.saarisalo thanks for the feedback and for your patience in this major effort to make our runners safer. There has been a change in the planned roadmap in the meantime (as per the description in this issue): we're preserving the register command but allowing it to accept a runner token in place of a registration token. That will allow the best of both worlds while requiring much less effort on the part of the users. The message on the gitlab-runner register command has been adapted accordingly and will be visible in %15.7 if you use a registration token with the register command.
Thanks @pedropombeiro for your response. I fully appreciate the efforts to make runners safer and better!
My observation and the only criticism here is that the way 15.6.1 behaves looks odd and confusing for the users who do not have prior information about the ongoing improvement plans. gitlab-runner!3742 (diffs) also says "has been deprecated..", whereas I believe it would be better to say "will be deprecated..".
The difference is that "has been deprecated" implies a corresponding, better means is available and should be used instead of the deprecated functionality. It'd be more clear to talk in future tense until the better way is actually available and the user can act upon the information.
Normally we specify the registration token using --registration-token "INSTANCE_REGISTRATION_TOKEN". But now we're getting "Failed to register runner" with the the following error: WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens
The docs linked above still say to use the --registration-token flag and don't offer an alternative. Does that mean one-line registration is broken at the moment?
Hi @binaryfire. The runner is not supposed to fail registration at the moment if you use a registration token. There is only a warning message to bring attention to the fact that there is an upcoming change to the behavior. Can you send a screenshot of the full output of the register command?
Hi @pedropombeiro. Apologies, it seems to be working now.
Our Ansible playbook updates both Gitlab and the runner. We unregister then re-register the runner on each playbook run. For some reason the unregister command didn't work properly directly after updating to Gitlab 17.2.
I removed the runner manually via the admin interface and everything is working again (including the unregister command)
Hi @pedropombeiro,
I am sorry we are getting error along with warning as below while registering the runner. Can you pl suggest what could be issue?
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see #380872 (closed)ERROR: Registering runner... failed runner=build status=422 Unprocessable Entity
PANIC: Failed to register the runner.
Hi @kishorgovind. I don't think that the 422 error is related to this deprecation. That warning message is printed anytime you register a runner. Maybe you should open a separate bug report for this. From what I can see in the endpoint that registers a runner, only 400 Bad Request and 403 Forbidden should be returned by the endpoint. Maybe you can see more details in the GitLab logs or the Runner logs about the 422 error?
@m_schulze If the forks are not moved forward to incorporate the new changes, then it will stop working in %16.0. For self-managed GitLab instances, you will have the ability to enable a feature flag that will allow you to continue accepting the deprecated registration method until %17.0, at which point the feature flag and the functionality will be removed.
Under the assumption that the forks are moved forward, things will remain supported? I'm asking because of the possibility to introduce a handshake between the gitlab server with some kind of secret put into the gitlab-runner executables which would not be available for forks.
If the forks move forward, there should be no surprises. The major change on the runner side is the generation of a persistent system ID and sending it to the GitLab instance when polling for jobs.
I am assuming that the old registration will continue to function for some time, i.e. there will be overlap? When will the new functionality be introduced and when will the old one be removed? There needs to be overlap to allow time to make the change.
@mugane good question. In the architecture blueprint, we're planning on having the new registration method available in GitLab %15.10 alongside the existing registration method. The existing registration method will then be disabled by default in %16.6 (although it can be enabled until %17.0 on self-managed GitLab instances). Hope this addresses your doubts, otherwise I'll be happy to work with you to clarify things.
I'm really confused by this communication. Why does the warning redirect to this issue when the issues do not provide a straightforward explanation about the new process? Why am I still being directed to run an obsolete command in GitLab 15.8? Why is the blueprint so general?
Thanks for the feedback @timschneidersdk! In hindsight, it is clear that we could have made a better job with the communication. Hopefully, we can still improve the messaging so that users can have an easier time navigating the incoming changes.
Why am I still being directed to run an obsolete command in GitLab 15.8?
Are you referring to this warning (taken from the v15.8.1 tag). This warning is up-to-date. The command itself is not deprecated, as you can see in the title of this issue. What will disappear is the ability to use a registration token with it (replaced by an authentication token) and the ability to specify runner parameters, such as --active, --tag-list, etc.
Why does the warning redirect to this issue when the issues do not provide a straightforward explanation about the new process?
This issue should indeed have a short overview of the upcoming changes given that it is the landing page for the warning in the GitLab Runner. I've added an overview to the description, hoping that it will help clarify things for users. Please let me know if there are any concerns or improvement suggestions.
Why is the blueprint so general?
Do you have an area specifically that you feel is missing detail?
I agree, it is still unclear - for example, the Runner authentication token page does not say anything at all about how to use the new method to register a runner. It would be ideal to see an example of the actual commands (side-by-side) comparing the current method to the new method. Please note that this needs to be done in a non-interactive manner; in our case it generally scripts that will be doing all the registrations, no human input will be possible.
I agree, it is still unclear - for example, the Runner authentication token page does not say anything at all about how to use the new method to register a runner.
Great point @mugane! I will open an MR to update that part of the documentation.
Please note that this needs to be done in a non-interactive manner; in our case it generally scripts that will be doing all the registrations, no human input will be possible.
I would like to better understand your workflow. If in your existing workflow you create multiple runners with the same configuration for the purpose of being able to have multiple instances, then this would not be necessary in the new workflow. You would just create the runner once in the UI, and then use the authentication token shown in the calls to gitlab-runner register. They would all be the same runner to GitLab, but represented as different "runner machines" (which can have distinct IP addresses, architectures, etc.)
If instead you require creating completely different runners in an unattended fashion (for example with different tags, description, etc.), then you'd need to resort to an API call. We're working on a GraphQL API endpoint in the %15.9 milestone, and plan on working on the REST API endpoint on %15.11 at the latest.
I can't say anything about @mugane 's workflow, but I have recently automated setup of runners in our setup. We have several different runners (services) on one machine, they all have a slightly different configuration (it's mostly the docker part that is different), and also different tags so users of GitLab (we only have one instance - it's two load balanced servers, but I guess that should not pose additional problems) can choose (in general they don't know which tags are supported, they just know to tag the jobs in a specific way to get a runner that behaves like they have asked for.
(We also have other runners but they have a one-to-one relationship with the physical machines).
Thanks for the description @hcgrove. Given that you need to set different tags on each runner, you won't be able to reuse them and will need the upcoming REST API to be able to create them.
I'm working on a Q&A section that I plan to add to the architecture blueprint and link from the description of this issue. Hopefully, this will make life easier for users confronted with this change. MR: !111674 (merged)
This is a very poorly communicated change. Is there a released replacement for the "deprecated" tokens? If not then I don't see how a feature can be deprecated, since typically deprecated means you should no longer use it (but without a replacement there is no alternative!).
Currently the top level runner docs (that I was using as a new gitlab user) are apparently deprecated
You are right @zacharysabin, and I totally own that. I started with a misconception of the process of deprecation when I embarked on this change, and in trying to give as much advance warning to users as possible ended up causing unnecessary confusion. Part of the problem was that we were initially shooting for a closer removal date (%16.0) which created a sense of urgency in getting the message out. Ideally, the message should have been "We will be deprecating the registration token in time for a removal in GitLab 16.0", rather than jumping the gun on the deprecation process. I know I will have much more context on the next deprecation process that I go through, that's for sure
Enter the registration token:
ss
Enter a description for the runner:
[550db4e8b212]: s
Enter tags for the runner (comma-separated):
s
Enter optional maintenance note for the runner:
s
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see #380872 (closed)
Hi @kapilyadava.isa. Thanks for getting in touch. Once you create a runner in the GitLab admin UI (something we're currently working on), you will get an authentication token that you can use in place of the registration token when asked. We plan to provide a complete registration command in GitLab admin UI to make it easier to follow. You can find a detailed description and a diagram of the new flow in the architecture blueprint.
@pedropombeiro this whole change is super confusing to me. I don't seem to be able to register a runner via UI (when I try that I only get the info on how to do that with the old-style CLI commands). I get this info both as a user in the project/group and as GitLab admin in the corresponding admin UI. When trying to run the old-style CLI commands, I the get warning mentioned above and nothing happens.
Am I currently not able to register new runners at all until the admin UI change is done? Or am I missing something?
Ah, my bad. I can register instance runners but not group runners. When I try to register a runner for a group, GitLab claims the token is wrong (which is not the case). So that looks like a different problem then.
Ok, found a solution. As GitLab Admin I had to enable the settings "Members of the project can register runners" and "Members of the group can register runners" in Settings > CI/CD > Runner registration
Funny enough, users are not able to access the /runners page of the group but the Admin can. So I could copy the command to create a runner and it worked.
How will this impact the GitLab Runner Helm Chart?
I'm currently very happy to be able to deploy/upgrade runners via helm/Argo CD using that helm chart.
Hi @fredde.lindgren, there is a related deprecation issue for Helm chart here: #381111 (closed). Basically, you'll need to create a runner in the GitLab runners admin UI, get the token from that and use it as the runnerRegistrationToken.
Thanks for the link. Is there any way to register runners without having to click around in the UI?
Would love to automate that process by creating a runner and fetching the token via an API.
@fredde.lindgren there is an MR in review to add a REST API for it: !116119 (merged). Please be aware that you should consider reusing a runner if the configuration doesn't change (tag list, paused status, etc.)
@pedropombeiro - Cool thanks!
How will this affect already registered runners?
Will they keep working as they've done previously since this only involves the register process?
Also we are quite happy at the moment with handling runner config via IaC, does this prevent us from doing this in any way?
Will they keep working as they've done previously since this only involves the register process?
Correct.
Also we are quite happy at the moment with handling runner config via IaC, does this prevent us from doing this in any way?
@fredde.lindgren It shouldn't because the user-facing scope of the change is rather small, but can you go a bit into more depth into how you're handling the runner config?
trying to use runnerRegistrationToken but I get the following error when using the registration token:
FATAL: Runner configuration other than name, description, and executor configuration is reserved and cannot be specified when registering with a runner token. This configuration is specified on the GitLab server. Please try again without specifying those arguments.
Did not have this with the previous generic registration key.
Sorry for a very delayed response.
We use gitops with an internal fork of the gitlab runner helm chart (basically works the same but with some default values set and using non-root images).
In git teams then set chart versions to install via gitops along with values for things like tags, run_untagged, register_locked, destination namespace in openshift, resources for main/job pods, etc.
This way we got the configuration for the runners saved as IaC.
I can only highlight that these changes make manual actions in the UI necessary, unless gitlab provides this as an automated process through the runners somehow (which would make this change non intrusive).
As @fredde.lindgren mentioned, IaC and GitOps works just fine with the static registration token, and will become a hassle with these changes. I think this is a step back from IaC+GitOps and towards more ClickOps
I'm with @florian.berchtold on this. Requiring you user to engage with APIs to manage this is a step back for already over-burdened ops teams. I can appreciate the security improvements this brings, but security is part of a story that needs to be told in a way that user's are enabled and empowered, instead of disgruntled.
Thanks @DarrenEastman for the link. Deploying static runners is one way to do it. Trivial, but expensive and manual to scale though.
As you know, we utilize the Kubernetes runner manager, to utilize the autoscaling capabilities of Kubernetes together with Karpenter. We have a templated helm chart, and to add a new runner, all we need to do is copy a few lines in the values.yaml, modify some (like when adding a runner with more CPU resources, just increase the number that represents the requested cpu resources for the POD). With registration tokens, we can reuse the existing one multiple times and create new runner managers. I.e. only a single time an admin is required to do something on the UI.
If you compare this to the new workflow, you will need an additional building block that needs to use a stored secret, the authentication token, run it against the API to retrieve a registration token, and create a new secret that can be found by the manager runner that got deployed by the helm chart. Will GitLab provide this controller?
So Gitlab seems to only have the trivial and tutorial-ish use cases in mind, and misses out on production grade deployments when coming up with these kind of changes?
@DarrenEastman Does Gitlab provide a solution for real-life production use cases, with a scale that requires more than a hand full of static provisioned runners?
No, we do not use Terraform directly, but utilize crossplane for IaC. How would this API be used outside of Terraform, can you provide a working example?
Have I understood the new method correctly when it seems to make automated registration of new runners harder? Requiring manual work in the UI - bad idea!
It's already a pain that the current process starts with "User retrieves runner registration token in GitLab UI", but that mostly causes us grief in test environments, in production we can just retrieve that token and store it securely in our configuration management system for that to use when deploying runners.
It seems like the new procedure starts "User creates a runner in GitLab UI and adds the runner configuration", will there be an API call to do that? Or some other way to automate it?
For production we would like people who don't normally administrate GitLab to be able to just add a new server to our configuration management (still far from every user of GitLab who can do that) and have that configured and set up as a runner a little later.
For testing we would like to be able to spin up two new "servers", one running GitLab and one running gitlab-runner, and have the new runner automatically register against the new GitLab instance, that's hard even with the current workflow (I'm trying to make something work with the reset token API call that also outputs the new, then I can make the runner server issue that API call and use the received token to register a new runner)
This is beginning to look like another case of the administrator hostility that I have found several places in GitLab.
Requiring manual work in the UI - bad idea! It's already a pain that the current process starts with "User retrieves runner registration token in GitLab UI"
@hcgrove Not sure if you read the new registration process, but in that process there is no more user interaction in the new process than what is required right now. Instead of going to the UI to fetch the runner registration token, you go there to create a runner (in this context, it can be thought of a configuration) and then use the runner token that is shown wherever you are currently using the registration token. The difference is that previously you could use the same registration token to create runners with any configuration, whereas now you must pre-create the configurations (which results in better security).
will there be an API call to do that?
There is already a GraphQL mutation for it, and there is an issue to create the equivalent REST API.
@hcgrove Not sure if you read the new registration process, but in that process there is no more user interaction in the new process than what is required right now. Instead of going to the UI to fetch the runner registration token, you go there to create a runner
But the runner registration token needed to be fetched once, whereas the new stuff needs to be done for each runner, it is more work (if you have more than one runner, but my guess is that many installations have). And as it was once, it wasn't manual work for each runner.
Furthermore there is more work in entering entering configuration than in retrieving a token, so there is more work.
So my conclusion still is that the new method requires moremanual work for each runner, than the old.
And it doesn't give better security if you knew what you were doing and kept the old registration token safe. So this is a case of an advantage on expense of those who knew what they were doing, not a sound development strategy (but unfortunately one widely seen).
So it's moremanual work for the benefit of some (I even doubt it's a majority, those maintaining an instance often know what they are doing).
There is already a GraphQL mutation for it,
I should look into GraphQL then.
there is an issue to create the equivalent REST API.
I wish GitLab would make it a habit to make sure REST API calls exist (and are documented) before features are released, at least until GraphQL is ready to take over as the primary means of communication (the vision according to your documentation on the GraphQL API).
But the runner registration token needed to be fetched once, whereas the new stuff needs to be done for each runner, it is more work (if you have more than one runner, but my guess is that many installations have).
Just so I understand, what is the reason why you need to create a new runner? Is there anything fundamentally different in the new runner configuration from the runners you previously registered?
I wish GitLab would make it a habit to make sure REST API calls exist (and are documented) before features are released, at least until GraphQL is ready to take over as the primary means of communication (the vision according to your documentation on the GraphQL API).
The new registration workflow is not released yet, and according to the current implementation plan, the REST API should be there when the time comes to enable the feature flag that will expose the new workflow. The reason why the GraphQL version will always come first is because that is what the web app consumes (as it is much more efficient than consuming the equivalent REST API). But I totally hear you about GraphQL not yet covering the whole API surface that the REST API does. I know there is an internal initiative to bring them to par with each other.
what is the reason why you need to create a new runner?
Some reasons are:
More employees (=> more projects => more CI/CD jobs)
Hardware replacement
More requirements from colleagues on what should be possible in a runner (e.g. our docker runner didn't support docker-in-docker setups, but somebody wanted it, so we set up a new runner that allowed that)
Stricter security requirements (we have projects that want their CI/CD jobs not to have access to the internet)
Is there anything fundamentally different in the new runner configuration from the runners you previously registered?
If met with new requirements there might be, but in most cases: No, just a wish to automate it more. But that doesn't mean I won't need to deploy new runners and be affected by the changes to the procedure.
according to the current implementation plan, the REST API should be there when the time comes to enable the feature flag that will expose the new workflow
Unfortunately implementation plans can be deviated from, but apart from that it sounds good.
But there is only one task in that stage mentioning the REST API (#390427 (closed)), and that task has "Develop a REST API for creating a Runner using an authentication token." as it's proposal, and it's the creation of such a token I'm talking about (I also don't like the "Adapt" in the title of that task, as it is completely new functionality that is needed, but that is hopefully just a meaning of "adapt" I don't know).
If met with new requirements there might be, but in most cases: No, just a wish to automate it more. But that doesn't mean I won't need to deploy new runners and be affected by the changes to the procedure.
If the configuration remains the same, nothing prevents you from reusing the authentication token (how you keep these runner authentication tokens securely is up to you), removing the need to resort to automation to create a new runner. The runner that is registered in a new machine would automatically get a new system ID visible in the build logs.
But there is only one task in that stage mentioning the REST API (#390427 (closed)), and that task has "Develop a REST API for creating a Runner using an authentication token." as it's proposal, and it's the creation of such a token I'm talking about (I also don't like the "Adapt" in the title of that task, as it is completely new functionality that is needed, but that is hopefully just a meaning of "adapt" I don't know).
That issue is the right placeholder, unfortunately the details are incorrect. The idea is to have an authenticated endpoint where you can create a new runner and receive an authentication token in return. This endpoint will only be available to users that have permissions to create new runners.
Just to summarize, currently it is only possible to create a runner+token in UI, but later we will get an runners API ebdpoint that allows us to create and fetch the tokens to automate this?
As well as checking if a machine by system id, token or name already was created resp. is still valid?
Just a remark if it was mentioned somewhere you could also implement clean-up strategies for runners (if they were stale for too long etc.).
Just to summarize, currently it is only possible to create a runner+token in UI, but later we will get an runners API ebdpoint that allows us to create and fetch the tokens to automate this?
Why did you address me there? I'm actually trying the REST API right now (first attempt resulted in a SSL error in my test setup, not really GitLab related), and actually considered writing a reply saying it was there, but chose not to because I didn't bother digging up the links, and (like you) didn't understand the second part.
But that is a special case that wouldn't need to be treated as a special case if registration of new runners was easy.
nothing prevents you from reusing the authentication token
But as far as I understand it that would probably make it hard to continue using the old runner (for a period), and remember that one of the use cases of registering a new runner was "More employees".
And in the case of "Hardware replacement": transitions can be smoother when you can have both the old and new running simultaneously for a short while.
removing the need to resort to automation to create a new runner
But that automation is wanted any way to handle all the other cases, and when automated creation is in place, it is much easier to use it in any case, than to detect when reuse is possible and divert to use that.
That issue is the right placeholder, unfortunately the details are incorrect.
I hope you're right.
This endpoint will only be available to users that have permissions to create new runners.
As it should be. I've not argued for lowering your security, just that increasing it should not make things harder.
But that is a special case that wouldn't need to be treated as a special case if registration of new runners was easy.
This is precisely an aspect we're trying to get under control, as today we have millions of "stale" runners that were created as a throw-away runner and never unregistered, which create unnecessary road-blocks to new functionality, and challenges in terms of scaling the product.
And in the case of "Hardware replacement": transitions can be smoother when you can have both the old and new running simultaneously for a short while.
Reusing the authentication token here would be a perfect scenario: the new machine just gets a new system ID and you can run them side-by-side for a while.
I hope you're right.
I'm changing the details of that issue to match the design as we speak.
As it should be. I've not argued for lowering your security, just that increasing it should not make things harder.
If the new workflow solves the needs of even half of the use-cases (such as runner managers running in a Kubernetes environment), this will represent a much lower number of runners overall, and make their management much more tractable in the long term. As a fallback, this is exactly the reason why we're keeping an API, to still allow use cases which are not addressed by the new workflow.
I am not sure where else to post this since this is linked to.
This deprecation seems to kill one of the handy features of the registration process...being able to specify tags. This also represents more than "deprecating one token for another" as these docs and issues imply.
We have different runners deployed via helm charts that have a set of tags defined depending on their environment. Having the tags defined before the registration process moves the tag definition to that phase. In order to restore the previous functionality one either needs one definition per tag combination and switch between the tokens. Or I guess use something like the terraform gitlab provider to create the runner on the fly and pass the token to the helm chart.
In the end this just makes what used to be trivial...non-trivial. Perhaps something can be improved about that or some proper migration docs written.
To be fully transparent, what is the harm in letting teams continue using the registration token? If some folks want to trade the flexibility of registration options for a stronger audit trail so be it. There is no inherit security risk since anyone with access to get the current token can create a new runner and get a token.
Currently the terraform provider does not support registering without a registration token and the registration token does not work is a personal/group access token is used. So this effectively isn't an option.
Thanks for the context @jimmy-outschool, I appreciate learning more about how users are using the product.
I am not sure where else to post this since this is linked to.
There is a feedback issue available: #387993 (closed), please feel free to repost your question there so others can benefit from the discussion.
This deprecation seems to kill one of the handy features of the registration process...being able to specify tags.
Maybe the terraform provider could mimic the same functionality by leveraging the POST /user/runners REST endpoint if there is a need to specify the tags dynamically. The downside is that these runners would not appear grouped under a single configuration, since effectively we'd be creating a new configuration every time even if the configuration has already been created before.
In order to restore the previous functionality one either needs one definition per tag combination and switch between the tokens.
I guess if you have 1 or 2 sets of tags you use, that would be the ideal solution, since you'd see all your runners grouped together by configuration, among other advantages like automatic cleanup of stale runners. But if you're dealing with dynamic configurations, creating a new runner configuration is the realistic approach.
To be fully transparent, what is the harm in letting teams continue using the registration token? If some folks want to trade the flexibility of registration options for a stronger audit trail so be it. There is no inherit security risk since anyone with access to get the current token can create a new runner and get a token.
There is the non-negligible aspect of the added complexity in the codebase and respective drag on any future architecture decisions. Leaving a parallel concept inside the software doesn't come for free, unfortunately.
@ImryLevySadan you can use the REST API mentioned in #380872 (comment 1421905433) to create the runner configuration you need. This will get you back a runner authentication token that you can use in place of the registration token to register the runner.
But I think the real question is is there a way to deploy gitlab in a fully programmatically way, with no manual action in the process?
The scope of this question is the IaC world, were gitlab instance and runners are being quite easily deployed with tools such as Terraform or Pulumi.
I thinks there is, because PAT can be created manually, and after that I can create runners with api, get the runners token and pass it into my registration proccess.
But it looks like Gitlab hide this flow and does not recommend it, mainly, for security issues
@ImryLevySadan I'm not sure what are the missing pieces for it on your end, but FYI there is an open issue on the Terraform provider repo to add a resource to create a runner: terraform-provider-gitlab#1428 (closed).
Using REST API seems to be a big step backwards regarding security. Now, I have to publish a security token with admin API access permissions while previously only a token with limited range was used.
Now, I have to publish a security token with admin API access permissions while previously only a token with limited range was used.
@ensc what security token are you referring to? If you're referring to the PAT, that only has permissions to create a runner, due to the new create_runner scope.
I had some confusion about the change that I think I've seen reflected in other comments, which this earlier reply helped clarify for me. I've also read through the blueprint which helped as well.
To summarize how my team uses the runner feature, we use terraform to spin up X number of instances that each gets passed the group runner registration token and runs the deprecated one-liner registration command, and afterwards each instance shows up as an individual runner in the GitLab group runner UI.
Thus, I think in terms "one runner, one instance" which is pretty jarring with the change to having one "runner" and ci_runner_machines. The term "runner" seems more like a "runner configuration" to me now. I don't think the registration flow change would result in too much of a difference with our actual setup process, since if I'm understanding right we would:
Manually create a single runner in the UI
Pass that runner's token to each instance
Run the new one-line registration command in each instance
We end with one runner with X runner machines under it
However, the management of these instances is still unclear to me, and I can see an issue that will come up because of how we use tagging.
For management, currently through the UI I can see lots of details for each runner including host information, description, etc. In addition, we can pause the runner. How much of this is visible with the registration flow change? I still want to have an easy way to pause individual machines from accepting jobs or see a description for each machine.
For tagging, we use tags to block newly-created instances from being able to take on jobs since, unfortunately, we need to perform a single manual step to finish setup. After that step we change the tags to allow the runner to take on jobs. We tear down and bring up all of our instances fairly often, so to minimize downtime we stagger this refresh process. Is there an elegant solution to doing this staggered instance refresh with the new registration flow?
Please feel free to correct me if I'm misunderstanding any part of the changes.
The term "runner" seems more like a "runner configuration" to me now.
@benjamin.wu1 Indeed, that is a helpful lens to look through.
For management, currently through the UI I can see lots of details for each runner including host information, description, etc. In addition, we can pause the runner. How much of this is visible with the registration flow change? I still want to have an easy way to pause individual machines from accepting jobs or see a description for each machine.
The plan is to show those details for the runner machines under each runner in the UI. Pausing individual machines won't be possible though, I'm afraid (only pausing a runner configuration).
For tagging, we use tags to block newly-created instances from being able to take on jobs since, unfortunately, we need to perform a single manual step to finish setup. After that step we change the tags to allow the runner to take on jobs. We tear down and bring up all of our instances fairly often, so to minimize downtime we stagger this refresh process. Is there an elegant solution to doing this staggered instance refresh with the new registration flow?
The tags only exist at the runner configuration level, so all runner machines share the same tags. Slightly unrelated, but why aren't you creating runners in paused state for that purpose?
Thanks for the response. And you're right about the pausing; I see an option for that in the gitlab-runner documentation, but forgot to mention that we do use them for one of their prescribed purposes: controlling which jobs the runners are allowed to run. It just so happens to be that it also coincides with the manual step we need to perform. I wasn't the original engineer behind this system, but this tag change step is called out in our documentation along with the manual step needed, so I suspect it helps remind whoever's doing the work to perform the step, whereas just unpausing might not feel as deliberate as changing a tag.
I request beg that this change include a tangible set of examples of how runner registration can be performed non-interactively.
This change will break Ansible playbooks that we use to manage our GitLab Runners. Perhaps future versions of these playbooks would preceded by invoking a GitLab API rest call to get a unique registration token for each runner host, prior to running the registration command? Regardless, I'm quite concerned about how we're going to continue to automate and more importantly normalize our runner configuration with the new workflow. The requirement of doing anything manually in the WebUI doesn't scale to our relatively small Premium context (200 licensed seats).
Perhaps future versions of these playbooks would preceded by invoking a GitLab API rest call to get a unique registration token for each runner host, prior to running the registration command?
The Ansible playbooks will need updates to invoke the new workflows, but everything can still be automated. First, retrieve the registration token via API call, then register the runner. This blog post explains all steps: https://about.gitlab.com/blog/2023/07/06/how-to-automate-creation-of-runners/ Updating the docs for automation steps is tracked in this issue #423020 Thanks @fneill for taking action.
@dnsmichi Is registration idempotent? What happens if we run the registration command (with the same token) twice on the same machine? Will a new entry be added under that runner in the UI or will the exiting entry with a matching ip address be updated / overwritten?
Also, how are stale entries handled now? Are they automatically removed from Gitlab after a certain period of time?
Is registration idempotent? What happens if we run the registration command (with the same token) twice on the same machine? Will a new entry be added under that runner in the UI or will the exiting entry with a matching ip address be updated / overwritten?
As long as the value in the .runner_system_id file located next to config.toml is the same, then the registration command will not create a new entry, and will just update the existing one.
Also, how are stale entries handled now? Are they automatically removed from Gitlab after a certain period of time?
@pedropombeiro I'm trying to figure out how to make the best use of 3 bare metal machines that'll be dedicated to hosting runners. I'm not using any container orchestration- they're just simple Ubuntu boxes. Could you help clarify a couple things? I'm a bit confused.
Concurrency
I thought concurrent set the ceiling on how many jobs all runners on a machine could handle, and that a single runner could handle multiple unrelated jobs. I.e. having a single runner on an 8-core machine with concurrent = 8 would be identical to creating 8 separate runners on the machine. But this example from the docs suggests otherwise?
Reusing a runner config with a simple setup
The runners on my 3 machines would all be identical. Same executor, tags etc. Can I use a token created in the new UI and to provision all of them? The docs about token reuse mention a 'runner manager' but that only seems relevant for Docker machine / k8s (neither of which I'm using).
3 x 8 core machines, each with a single runner installed and concurrent = 8, provisioned using the same token. That would give my instance the ability to run 24 concurrent jobs in total. And any of the 3 runners which aren't at capacity (i.e. which are running less than 8 jobs) would pick up new jobs as they're added to the queue.
So all 3 machines would have this in their config.toml and they would all three share the same token. But they each would generate a unique system_id when they first contact GitLab so they will appear as three "runners" grouped in a single entry in the UI. Each of them will pick up to 8 jobs at a time. They don't need to coordinate at all.
With the shell executor in a [[runners]] entry, the jobs will just be run locally. However for other executors, such as docker-machine, Kubernetes, docker-autoscaler (the Docker Machine replacement) and instance, the jobs will be run on other machines. So the overall concurrency will probably by much higher (representing the maximum capacity that autoscaling can achieve). But I don't think that's your situation, so you don't need to worry about it.
If anything is still unclear, post back and mention me! :)