## Problem At the moment we have a disjointed user experience in what a user is able to configure with the runner [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) and what we allow the user to configure with the [`values.yml`](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/48c832ad7e8975e5ba7e9dbdad192607d85b7fe8/values.yaml#L105-300). Every time we add a new configuration field in the Runner, we have to add support for that field in the helm chart, which increases the engineering workload. Apart from the extra work the maintainers have to do in order to support configuration in the helm chart, it's not always possible to use environment variables/flags to set the registration such as https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/83. This is why we created the [configuration template](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1263) so that we can specify a configuration template. ## Proposals Keep the `values.yml` as it is and just direct the user to specify their own [configuration template](https://docs.gitlab.com/runner/register/#runners-configuration-template-file). This would mean that for anything defined in the values.yml would have to rely on flags/environment variables. What we can do is to clean this up incrementally: 1. Introduce the configuration template, freezing anything new under the [`runners`](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/5b1af3e0bd52459aee5b0ff7d9752c2b569cafeb/values.yaml#L105), something similar to what we did in https://gitlab.com/gitlab-org/charts/gitlab-runner/-/commit/717506c71cc7b5f20409cc28d54a37bf2d3b87f6 1. Direct all users to use the configuration template, by updating the [docs](https://docs.gitlab.com/runner/install/kubernetes.html) 1. Deprecate the [`runners`](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/5b1af3e0bd52459aee5b0ff7d9752c2b569cafeb/values.yaml#L105), if it's possible print a warning message to the user if they are still specifying these values. 1. In [GitLab Runner chart 1.0](https://gitlab.com/groups/gitlab-org/charts/-/epics/12) we remove `runners` and have the user completely rely on the configuration template. So to give an idea of how the `values.yml` would look like it would be something like: ```yml runners: config: template: | [[runners]] output_limit = 1026 [runners.cache] Type = "s3" Path = "path/to/prefix" Shared = true [runners.cache.s3] ServerAddress = "s3.amazonaws.com" AccessKey = "AWS_S3_ACCESS_KEY" SecretKey = "AWS_S3_SECRET_KEY" BucketName = "runners-cache" BucketLocation = "eu-west-1" Insecure = false ``` #### Pros Assuming that we merge https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/197 and reject https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/153 1. There is no work for the maintainers where they add a configuration to the `gitlab-runner` project and would have to spend time translating it to the helm chart. 1. It gives full control to what the user wants. 1. If we add new configuration to the Runner the user has instant access to it without us having to push and update to the helm chart. #### Cons Assuming that we merge https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/197 and reject https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/153 1. We would have to stick to using flags/environment variables for anything we want to override which might not be possible for certain configuration. 1. Some keys might need to be duplicated since helm chart updates deployment depending on certain configuration for example [cache](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/3c88c85768fc8fa1f813bb0fbdec85a9955d7964/templates/deployment.yaml#L115-137), however this might be fixed if we translate the `toml` to `yaml`. <details> <summary> Rejected proposals </summary> ### Option 1: Go ahead with https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/153 This is what feels most natural to users of a helm chart, they just specify things in the `values.yml` and then the Runner team will handle the [configuration template](https://docs.gitlab.com/runner/register/#runners-configuration-template-file), go ahead with this we would be able to support [volumes](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/83) so the user just have to specify things in the `value.yml`. If we go ahead with this we would need to not merge and close https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/197 and https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/228 since as explained earlier we can't use the configuration template in two places. #### Pros Assuming that we merge https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/153 and rejecting https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/197 / https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/228 1. The user is just exposed to the `yml` configuration and not `toml`. It seems more natural to use `yaml` in the kubernetes ecosystem. 1. The user doesn't have to understand how [configuration templates work](https://docs.gitlab.com/runner/register/#runners-configuration-template-file) 1. This will allow us to add support for any type of configuration inside of the `values.yml` #### Cons Assuming that we merge https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/153 and rejecting https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/197 / https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/228 1. The maintainers of the helm chart are the bottlenecks of adding new features. So users would have to wait for us to add support to something. 1. We have to keep translating yaml to toml which is not always straight forward. #### Rejected reason I think we all agree here that this is less than ideal because of the cons related to it and it still doens't allow users to specify any configuration for the runner which doesn't improve the current situation that we are in. ### Option 3: Translate yaml to toml We would just translate yaml to toml using the `toToml` function available so that the user just has to write yaml and it is automatically translated to toml. The generated toml will be used as the configuration template #### Pros 1. Users just use yml which is what they are used too. 1. Everytime a new configuration is added we don't need to do anything. #### Cons 1. We might translation issues between yaml and toml (need to validate this) #### Rejected reason As tested in https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/106#note_348898590 this won't work because of the float/int translation issue. ### Option 4: Hybrid We can go a hybrid approach where we can use https://gitlab.com/gitlab-org/charts/gitlab-runner/-/merge_requests/153 but then if the user specifies the configuration template we leave it up to the user to configure everything. #### Pros #### Cons 1. We end up in a **very** weird state where something are defined in the `values.yml` and some aren't. There is no single source of truth the user would have to understand if they need to specify something in the `values.yaml` or in the configuration template setting. The only way to get around this is to have a breaking change that all configuration should be done inside of the configuration template. This however w 1. We have to explain how the merging of [configuration template](https://docs.gitlab.com/runner/register/#runners-configuration-template-file) works and how it merges things. 1. Anything that is defined in the values.yml will override the configuration template which might be confusing for the user. 1. Even more confusing why certain values from the `values.yml` aren't working because I have the configuration template defined. #### Rejected reason Has no pros, a lot of cons and makes it quite complex and doesn't fulfill the goal that the user can define everything. ### Option 5: Support multiple [configuration templates](https://docs.gitlab.com/runner/register/#runners-configuration-template-file) We can add support for the runner where you can specify multiple configuration templates and then we merge them in a specific order. This will be very confusing and hard to debug/develop on. #### Rejected reason Adds complexity that we would have to figure out merging and will cause a lot of issues over which template takes precedence. ### Option 6: Have gitlab-runner read yml configuration Update the [`gitlab-runner configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/fc6f8d606f95d90ac35f2b4876b68907c645dc37/common/config.go) read a `yaml` file in addition to `toml` so that we can just use the `values.yml` struct and save it on disk to for the runner to read. #### Pros 1. Users would be used to writing `yaml` if a kubernetes environment and they don't have to be exposed to yet another configuration language. 1. We don't have to translate `yaml` to `toml` or the other way round. 1. Both the helm chart and the `gitlab-runner` can use a common structure with no translation or duplication. #### Cons 1. Yet another configuration structure for `gitlab-runner` to support, making sure we check both a `config.toml` and `config.yml` file. Which increases the maintainability cost of `gitlab-runner` 1. It might be the case a tmol struct doesn't translate well to yml for existing configuration but this needs to be checked. </details> ## Issues that will be unblocked - https://gitlab.com/gitlab-org/charts/gitlab-runner/merge_requests/39 / https://gitlab.com/gitlab-org/charts/gitlab-runner/issues/83 - https://gitlab.com/gitlab-org/charts/gitlab-runner/issues/108 - https://gitlab.com/gitlab-org/charts/gitlab-runner/merge_requests/194#note_256109473 - https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/75