Next GitLab Runner Token Architecture (parent epic)
## Overview
The [GitLab Runner Token Architecture Evolution](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/) aims to introduce new runner registration and authentication mechanisms to simplify operational management and improve security.
## Implementation plan (Revised 2023-02-16)
### TLDR
- Disable the registration token by default (via feature flag) for all installations in 16.6 (including GitLab.com).
- Customers on self-managed GitLab can re-enable the legacy `registration token` method via a feature flag. This will be valid for a limited time period.
- Remove the feature flag. Target is approximately 17.0.
- Fully remove the legacy `registration token` method. Target = 17.0
### Stage 1 - Deprecations
<details>
<summary>
Completed
</summary>
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Rails app | %15.6 | <https://gitlab.com/gitlab-org/gitlab/-/issues/379743> | Deprecate `POST /api/v4/runners` endpoint for %17.0. This hinges on a [proposal](https://gitlab.com/gitlab-org/gitlab/-/issues/373774) to allow deprecating REST API endpoints for security reasons. |
| GitLab Runner | %15.6 | <https://gitlab.com/gitlab-org/gitlab/-/issues/380872> | Add deprecation notice for `register` command for %17.0. |
| GitLab Runner Helm Chart | %15.6 | <https://gitlab.com/gitlab-org/gitlab/-/issues/381111> | Add deprecation notice for `runnerRegistrationToken` command for %17.0. |
| GitLab Runner Operator | %15.6 | <https://gitlab.com/gitlab-org/gitlab/-/issues/382077> | Add deprecation notice for `runner-registration-token` command for %17.0. |
| GitLab Runner / GitLab Rails app | %15.7 | <https://gitlab.com/gitlab-org/gitlab/-/issues/383341> | Add deprecation announcement for registration token reset for %17.0. |
</details>
### Stage 2 - Prepare `gitlab-runner` for `system_id`
<details>
<summary>
Completed
</summary>
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Runner | %15.7 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28912> | Ensure a sidecar TOML file exists with a `system_id` value.<br/>Log new system ID values with `INFO` level as they get assigned. |
| GitLab Runner | %15.9 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28913> | Log unique system ID in the build logs. |
| GitLab Runner | %15.9 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29542> | Label Prometheus metrics with unique system ID. |
| GitLab Runner | %15.8 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29510>| Prepare `register` command to fail if runner server-side configuration options are passed together with a new `glrt-` token. |
| GitLab Runner | %15.9 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29525> | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29525+> |
| GitLab Runner | %15.9 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29592> | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29592+> |
### Stage 2a - Prepare GitLab Runner Helm Chart and GitLab Runner Operator
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
|GitLab Runner Helm Chart| %15.10|<https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/436>|<https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/436+>|
|GitLab Runner Operator| %15.10|<https://gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator/-/issues/148>|<https://gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator/-/issues/148+>|
| GitLab Runner Helm Chart | | <https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/455>|<https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/455+> |
</details>
### Stage 3 - Database changes
<details>
<summary>
Completed
</summary>
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Rails app | %15.8 | <https://gitlab.com/gitlab-org/gitlab/-/issues/386491> | Create database migration to add columns to `ci_runners` table. |
| GitLab Rails app | %15.8 | <https://gitlab.com/gitlab-org/gitlab/-/issues/386493> | Create database migration to add `ci_runner_machines` table. |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/386755> | Create database migration to add `ci_runner_machines.id` foreign key to `ci_builds_metadata` table. |
| GitLab Rails app | %15.8 | <https://gitlab.com/gitlab-org/gitlab/-/issues/386712> | Create database migrations to add `allow_runner_registration_token` setting to `application_settings` and `namespace_settings` tables (default: `true`). |
| GitLab Rails app | %15.8 | <https://gitlab.com/gitlab-org/gitlab/-/issues/387300> | Create database migration to add `config` column to `ci_runner_machines` table. |
| GitLab Runner | %15.8 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29390> | Start sending `system_id` value in jobs REST API requests that cause a runner heartbeat. |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/387398> | Create service similar to `StaleGroupRunnersPruneCronWorker` service to clean up `ci_runner_machines` records instead of `ci_runners` records.<br/>Existing service continues to exist but focuses only on legacy runners. |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/390261> | [Feature flag] Rollout of `create_runner_machine`. |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/387181> | Use runner token + `system_id` JSON parameters in jobs REST API requests in the [heartbeat method](https://gitlab.com/gitlab-org/gitlab/blob/c73c96a8ffd515295842d72a3635a8ae873d688c/lib/api/ci/helpers/runner.rb#L14-20) to update the `ci_runner_machines` cache/table. |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/389269> | [Feature flag] Enable runner creation workflow (`create_runner_workflow`) |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109531> | Implement `create_{instance\|group\|project}_runner` permissions |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/390241> | Rename `ci_runner_machines.machine_xid` column to `system_xid` to be consistent with `system_id` passed in APIs |
| GitLab Rails app | %15.10 | <https://gitlab.com/gitlab-org/gitlab/-/issues/391447> | Remove the ignore rule for `ci_runner_machines.machine_xid` column. |
| GitLab Rails app | %15.10 | <https://gitlab.com/gitlab-org/gitlab/-/issues/390858> | Replace `ci_builds_metadata.runner_machine_id` with a new join table |
| GitLab Rails app | %15.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/393340> | Drop `ci_builds_metadata.runner_machine_id` column. |
| GitLab Rails app | %16.0 | <https://gitlab.com/gitlab-org/gitlab/-/issues/392472> | Remove the ignore rule for `ci_builds_metadata.runner_machine_id` column. |
</details>
### Stage 4 - Create runners from the UI
<details>
<summary>
Completed
</summary>
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/383198> | Add prefix to newly generated runner authentication tokens. |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/389270> | Add new runner field for with token that is used in registration |
| GitLab Rails app | %15.9 | <https://gitlab.com/gitlab-org/gitlab/-/issues/387808> | Implement new GraphQL user-authenticated API to create a new runner. |
| GitLab Rails app | %15.10 | <https://gitlab.com/gitlab-org/gitlab/-/issues/391383> | Return token and runner ID information from `/runners/verify` REST endpoint. |
| GitLab Runner | %15.10 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29613> | Modify register command to allow new flow with glrt- prefixed authentication tokens. |
| GitLab Runner | %15.10 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29670> | Make the `gitlab-runner register` command happen in a single operation. |
| GitLab Rails app | %15.10 | <https://gitlab.com/gitlab-org/gitlab/-/issues/393887> | Define feature flag and policies for "New Runner creation workflow" for groups and projects. |
| GitLab Rails app | %15.10 | <https://gitlab.com/gitlab-org/gitlab/-/issues/394749> | Only update runner `contacted_at` and `status` when polled for jobs. |
| GitLab Rails app | %15.10 | <https://gitlab.com/gitlab-org/gitlab/-/issues/395013> | Add GraphQL type to represent runner machines under CiRunner. |
| GitLab Rails app | %15.11 | gitlab-org/gitlab#383139+ | Implement UI to create new instance runner. |
| GitLab Rails app | %15.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/393889> | Update service and mutation to accept groups and projects. |
| GitLab Rails app | %15.11 | gitlab-org/gitlab#383144+ <br> gitlab-org/gitlab#383143+| Implement UI to create new runner. |
| GitLab Rails app | %15.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/395783> | Add `runner_machine` field to `CiJob` GraphQL type. |
| GitLab Rails app | %16.1 | <https://gitlab.com/gitlab-org/gitlab/-/issues/388854> | UI changes to runner details view (listing of platform, architecture, IP address, etc.) (?) |
| GitLab Rails app | %15.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/390427>| Adapt `POST /api/v4/runners` REST endpoint to accept a request from an authorized user with a scope instead of a registration token. |
| GitLab Rails app | %"16.0" | <https://gitlab.com/gitlab-org/gitlab/-/issues/408206> | Handle glrt- runner tokens in `unregister` command. |
| GitLab Runner | %15.11 | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/30841> | Runner asks for registration token when a `glrt` runner token is passed in `--token`. |
| GitLab Rails app | %15.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/404744> | Move from 'runner machine' terminology to 'runner manager'. |
### Stage 4a - UI changes for runner managers
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Rails app | %16.1 | <https://gitlab.com/gitlab-org/gitlab/-/issues/409388> | New runner details page to support groups of runners. |
| GitLab Rails app | %16.3 | <https://gitlab.com/gitlab-org/gitlab/-/issues/421188> | Change filtering by runner upgrade status to rely on runner manager versions. |
</details>
### Stage 4b - API changes for runner managers
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Rails app | %16.2 | <https://gitlab.com/gitlab-org/gitlab/-/issues/415444> | Add `runner_manager` field to `API::Entities::Ci::Job`. |
| GitLab Rails app | %16.2 | <https://gitlab.com/gitlab-org/gitlab/-/issues/415923> | <https://gitlab.com/gitlab-org/gitlab/-/issues/415923+>. |
| GitLab Runner app | | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/33706> | <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/33706+>. |
| GitLab Terraform provider | %"16.3" | <https://gitlab.com/gitlab-org/terraform-provider-gitlab/-/issues/1428> | <https://gitlab.com/gitlab-org/terraform-provider-gitlab/-/issues/1428+>. |
| GitLab Rails app/GitLab Runner app | | <https://gitlab.com/gitlab-org/gitlab/-/issues/423523> | Add `description` column to `ci_runner_machines` table and populate it when runner is registered. |
| GitLab Rails app | %16.8 | <https://gitlab.com/gitlab-org/gitlab/-/issues/432586> | Create new REST/GraphQL queries to bring parity between runner managers and the existing queries for runners (i.e. be able to list jobs executed by a specific runner/runner manager combination). |
### Stage 5 - Optional disabling of registration token
<details>
<summary>
Completed
</summary>
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Rails app | %16.0 | <https://gitlab.com/gitlab-org/gitlab/-/issues/404804> | Adapt `register_{group\|project}_runner` permissions to take [application setting](https://gitlab.com/gitlab-org/gitlab/-/issues/386712) in consideration. |
| GitLab Rails app | %16.1 | <https://gitlab.com/gitlab-org/gitlab/-/issues/410680> | Make [`POST /api/v4/runners` endpoint](../../../api/runners.md#register-a-new-runner) based on a registration token permanently return `HTTP 410 Gone` if `allow_runner_registration_token` setting disables registration tokens.<br/>A future v5 version of the API should return `HTTP 404 Not Found`. |
| GitLab Rails app | %16.1 | <https://gitlab.com/gitlab-org/gitlab/-/issues/409389> | Add runner group metadata to the runner list. |
| GitLab Rails app | %16.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/411923> | Add UI to allow disabling use of registration tokens at top-group level. |
| GitLab Rails app | %16.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/448796> | Add UI to allow disabling use of registration tokens in admin panel. |
| GitLab Rails app | %16.11 | <https://gitlab.com/gitlab-org/gitlab/-/issues/411917> | Hide legacy UI showing registration with a registration token, if it disabled on in top-level group settings or by admins. |
| GitLab Runner app | %16.2 | <https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4157> | <https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4157+> |
</details>
### Stage 6 - GitLab Runner Identity Tokens
See design document: https://gitlab.com/gitlab-org/architecture/gitlab-runner-identity/design-doc
This will reduce the need to rotate runner authentication tokens and introduce a plugin model for token storage, thereby lessening the problem of race conditions between runners sharing the same token and requesting token rotation.
### Stage 7 - Enforcement
See in the dedicated epic: <https://gitlab.com/groups/gitlab-org/-/epics/10751+>
### Stage 8 - Removals
| Component | Milestone | Issue | Changes |
|------------------|----------:|-------|---------|
| GitLab Rails app | TBD | | Remove UI enabling registration tokens on the group and instance levels. |
| GitLab Rails app | TBD | | Remove legacy UI showing registration with a registration token. |
| GitLab Runner | TBD | | Remove runner model arguments from `register` command (for example `--run-untagged`, `--tag-list`, etc.) |
| GitLab Rails app | TBD | | Create database migrations to drop `allow_runner_registration_token` setting columns from `application_settings` and `namespace_settings` tables. |
| GitLab Rails app | TBD | | Create database migrations to drop:<br/>- `runners_registration_token`/`runners_registration_token_encrypted` columns from `application_settings`;<br/>- `runners_token`/`runners_token_encrypted` from `namespaces` table;<br/>- `runners_token`/`runners_token_encrypted` from `projects` table. |
| GitLab Rails app | TBD | https://gitlab.com/gitlab-org/gitlab/-/issues/453949 | Remove `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN`. |
## Pending questions
- [ ] Should `system_id` be logged in LabKit (`Gitlab::ApplicationContext`)?
- [x] Should runner machine ID be added to REST API jobs entity?
- [x] Should we expose runner machines on GraphQL and REST APIs?
- [ ] ...
## Related Token Rotation MR's
| MR| Description |
| ------ | ------ |
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77884+|Adds db settings for runner `authentication` token expiration intervals (instance, group, project)|
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77899+|Adds a database field `token_expires_at` to store the runners `authentication` token expiration time.|
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78557+|Enforces the expiration of the runner `authentication` token|
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78939+|Add sorting of CI runners table by `token_expires_at|
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78941+|Exposes API settings for runner token expiration|
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79135+|Adds GraphQL functionality to the runner `authentication` token expiration|
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75760+||
|https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3209+| adds a new reset-token subcommand, and automatically resets tokens as necessary in the run subcommand. This is a pre-req for the front-end merge requests listed above.|
|https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185544+|Adds service ping for reporting instances with runner token rotation settings enabled.|
## Workflow Diagram
{width="1500px"}
## Testing matrix
| Scenario | gitlab-runner version | GitLab instance version | `create_runner_machine` FF | Expected result |
| -------------------------------- | --------------------- | ----------------------- | -------------------------- |--------------- |
| `gitlab-runner register -t $REGISTRATION_TOKEN` | 15.6 | 16.0 | N/A | GitLab verifies that the registration token is valid. Should work correctly even though the endpoint now returns JSON and the caller expects plain/text. |
| `gitlab-runner register -t $REGISTRATION_TOKEN` | 15.10 | 15.8 | N/A | GitLab verifies that the registration token is valid. Should work correctly even though the endpoint returns plain/text and the caller expects JSON (runner will repeat request using `plain/text`). |
| `gitlab-runner register -t $REGISTRATION_TOKEN` | 15.10 | 16.0 | N/A | GitLab verifies that the registration token is valid. Since both the caller and endpoint exchange JSON, there should be no issues. |
| `gitlab-runner register -r $RUNNER_TOKEN` | 15.10 | 16.0 | true | GitLab verifies the runner with the `glrt-` prefixed runner token, creates first runner machine, and runner adds it to the `config.toml`. |
| `gitlab-runner register -r $RUNNER_TOKEN` | 15.10 | 16.0 | true | GitLab verifies the runner with the `glrt-` prefixed runner token, runner machine is not created, and runner adds it to the `config.toml`. |
| `gitlab-runner register -r $REGISTRATION_TOKEN` | 15.10 | 16.0 | N/A | Runner calls `POST /runners` to create a new runner. Since both the caller and endpoint exchange JSON, there should be no issues. |
epic