Configuration file template for registration command (!1263) · Merge requests · GitLab.org / gitlab-runner

Tomasz Maczukin requested to merge config-template-for-registration-poc into master Mar 26, 2019

What does this MR do?

Implements a PoC of configuration templates support for registration.

Why was this MR needed?

Some of configuration settings of Runner can't be set with env variables or command line options. Env variables are for example not supporting slices. And for some settings (e.g. the whole K8S executor volumes tree) we intentionally didn't provide even the command line options support.

This however becomes a problem in automatically manage environments (e.g. when using GitLab Runner's official Docker images or Helm Chart for Runner deployment). And in such cases the only solution for now is to manually update the config.toml file after the Runner was registered. This is hacky, this is error prone and this is definitely not reliable - especially when more than one registration for the same Runner installation is done.

This MR tries to solve this problem in a way that will be generic for all environments. The register command got a --template-config command line option (that can be also set with the TEMPLATE_CONFIG_FILE environment variable). When it's set it expects that a path to a configuration file template will be given as the value for this flag.

The configuration template file supports the subset of Runner's config.toml file: it's restricted to support only [[runners]] section and all it descendants. For the PoC implementation the global options are not supported.

Additionally the configuration template requires that only one [[runners]] section is present in the file.

When --template-config is used, then the configuration of [[runners]] entry is merged into the configuration of newly created [[runners]] entry in the regular config.toml file. The merging is done only for options that were empty (so empty strings, nulls/non existing entries, zeroes). With this all configuration provided with command line options and/or environment variables during the register command call take the precedence, and the template fills the gaps and adds additional settings.

Let's see a live example

First, let's try to register a K8S executor based Runner to some test project and see how the config.toml file looks like:

$ gitlab-runner register \
     --config /tmp/test-config.toml \
     --non-interactive \
     --url https://gitlab.com \
     --registration-token __REDACTED__ \
     --name test-runner \
     --tag-list kubernetes,test \
     --locked \
     --paused \
     --executor kubernetes \
     --kubernetes-host http://localhost:9876/

Runtime platform                                    arch=amd64 os=linux pid=1684 revision=88310882 version=11.10.0~beta.1251.g88310882
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Registering runner... succeeded                     runner=__REDACTED__
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

$ cat /tmp/test-config.toml
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "test-runner"
  url = "https://gitlab.com"
  token = "__REDACTED__"
  executor = "kubernetes"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.kubernetes]
    host = "http://localhost:9876/"
    bearer_token_overwrite_allowed = false
    image = ""
    namespace = ""
    namespace_overwrite_allowed = ""
    privileged = false
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.volumes]

We can see the basic configuration created from the provided command line options. There are Runner credentials (URL and token), the executor specified and the default, empty section runners.kubernetes with only the one option provided during the registration filled. Normally one would need to set few more options to make the K8S executor usable, but the above is enough for the purpose of our example.

Let's now assume that we need to configure an emptyDir volume for our K8S executor. There is no way to add this while registering with neither env variables nor command line options. We would need to manually append something like this to the end of the file:

[[runners.kubernetes.volumes.empty_dir]]
  name = "empty_dir"
  mount_path = "/path/to/empty_dir"
  medium = "Memory"

TOML doesn't require a proper indentation - it relays on entries ordering. So we could just append the required changes to the end of the file. However, this becomes tricky when more [[runners]] are being registered within one config.toml file. The assumption that the new one will be always at the end is risky.

With this MR, the solution however is simple. First, let's create a configuration file template:

$ cat > /tmp/test-config.template.toml << EOF
[[runners]]
  [runners.kubernetes]
    [runners.kubernetes.volumes]
      [[runners.kubernetes.volumes.empty_dir]]
        name = "empty_dir"
        mount_path = "/path/to/empty_dir"
        medium = "Memory"
EOF

Having the file, we can now try to register the Runner again, but this time adding the --template-config /tmp/test-config.template.toml option. Apart of this change, the rest of registration command will be exactly the same:

$ gitlab-runner register \
     --config /tmp/test-config.toml \
     --template-config /tmp/test-config.template.toml \
     --non-interactive \
     --url https://gitlab.com \
     --registration-token __REDACTED__ \
     --name test-runner \
     --tag-list kubernetes,test \
     --locked \
     --paused \
     --executor kubernetes \
     --kubernetes-host http://localhost:9876/

Runtime platform                                    arch=amd64 os=linux pid=8798 revision=88310882 version=11.10.0~beta.1251.g88310882
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Registering runner... succeeded                     runner=__REDACTED__
Merging configuration from template file           
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

As we can see, there is a little change in the output of the registration command. We can see a Merging configuration from template file line.

Now let's see how the configuration file looks like after using the template:

$ cat /tmp/test-config.toml 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "test-runner"
  url = "https://gitlab.com"
  token = "__REDACTED__"
  executor = "kubernetes"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.kubernetes]
    host = "http://localhost:9876/"
    bearer_token_overwrite_allowed = false
    image = ""
    namespace = ""
    namespace_overwrite_allowed = ""
    privileged = false
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.volumes]

      [[runners.kubernetes.volumes.empty_dir]]
        name = "empty_dir"
        mount_path = "/path/to/empty_dir"
        medium = "Memory"

We can see, that the configuration is almost the same as it was previously. The only change is that it now has the [[runners.kubernetes.volumes.empty_dir]] entry with its options at the end of the file. It's added to the [[runners]] entry that was created by the registration. And because the whole file is saved with the same mechanism, we additionally have the proper indentation.

Let's also check what would happen, if the configuration template file would contain a setting that is also being assigned by the registration command itself:

$ cat > /tmp/test-config.template.toml << EOF
[[runners]]
  executor = "docker"
EOF

$ gitlab-runner register \
     --config /tmp/test-config.toml \
     --template-config /tmp/test-config.template.toml \
     --non-interactive \
     --url https://gitlab.com \
     --registration-token __REDACTED__ \
     --name test-runner \
     --tag-list shell,test \
     --locked \
     --paused \
     --executor shell

Runtime platform                                    arch=amd64 os=linux pid=12359 revision=88310882 version=11.10.0~beta.1251.g88310882
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Registering runner... succeeded                     runner=__REDACTED__
Merging configuration from template file           
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

As we can see, the registration command is specifying the shell executor, while the template contains the docker one. Let's see what is the final configuration content:

$ cat /tmp/test-config.toml 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "test-runner"
  url = "https://gitlab.com"
  token = "__REDACTED__"
  executor = "shell"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

And as we can see, the configuration set with register command options got the priority and was chosen to be placed in the final config.

Are there points in the code the reviewer needs to double check?

This is a PoC. It's currently not proven, that it will be 100% reliable. The template mechanism uses https://github.com/imdario/mergo for merging the structs (the one taken from configuration options and the one loaded from template file).

In the library documentation we can find, that it - for example - doesn't support maps merging, because map can't be addressed with Go's reflection mechanism. Before making the configuration file template feature the officially supported one, we should check what other limitations the library has and if our configuration structure may be affected by them.

But for the purpose of PoC and testing this approach in real live I think the MR is ready to go :)

What's left to be done?

list all limitations of github.com/imdario/mergo
check if we are using any unsupported constructions in our configuration structs hierarchy
[-] decide what to do with such cases (if there are any)

Does this MR meet the acceptance criteria?

[-] Documentation created/updated - Will be created in a follow-up MR, please check #4228 (closed)
Added tests for this feature/bug
In case of conflicts with master - branch was rebased

What are the relevant issue numbers?

Reference #4228 (closed)

Edited Aug 06, 2019 by Steve Xuereb

Configuration file template for registration command