Manage Hosted Runners tokens through Terraform

As part of Transition Hosted runners on Linux for GitLab.c... (&17) • Tomasz Maczukin we want to migrate from hand-managing runner registrations of our Hosted Runners to registration of tokens managed through Terraform and direct storage in Vault. With the number of instances and configurations that we provide and limited (as it should!) access to admin permissions in our GitLab instances, that approach doesn't scale anymore. With upcoming Cells introduction it will be even harder.

Therefore we want to place that configuration in Terraform with changes applied through GitLab CI/CD using an admin Token with scope limited to just managing runners. The plan is that we will have only one token per shard (and will use the system_id feature to distinguish specific instances of the runner managers) so we will have much less places to manage than we have now (where, for example, for saas-linux-small-amd64 if we would like to change the cost factor, we need to update 12 ci_runners entries on GitLab.com)

Most of that is straightforward, but while working on a PoC a first question have arise: Should we manage cost factors through Terraform?

The problem is that cost factor settings are available in GitLab Terraform Provider. It’s not there as github.com/xanzy/go-gitlab doesn’t support that, and that is not supported primarily because GitLab REST API doesn’t support it. So at least three places to update first, before we will be able to handle it.

I’ve started working on the Rest API update as it’s the first thing to add anyway, but would like to hear other opinions as to whether it’s even worth the effort.

In my opinion we have three ways how we can move forward with this problem:

1. Do nothing. Start managing tokens through Terraform and then manually adjust cost factors through UI. For that an admin account is required so number of people who can do that is limited and for SREs and support engineers who don’t work with runners - they will need to be instructed each time. We can do it also through Rails console, but access here is also limited and adding production changes that way is risky and should be done as exception, not as a regular way of managing things. And it will get harder when Cells will be introduced. This approach is something I would like to avoid as much as possible.

2. Update just the Rest API and then use some other Terraform Provider that allows to interact with rest APIs, like for example mastercard/restapi, to make a direct HTTP call from Terraform and update cost factors on a runner that was created with gitlab_user_runners resource. This will be simpler change as it requires just update of our API and no other dependencies (especially given that the cost factor fields at this moment are usable only for instance runners on GitLab.com - that's hardcoded into GitLab - so no other user of GitLab Provider for Terraform nor go-gitlab library would benefit from it). It's not as clean on our Terraform code side, but totally doable and a fair compromise in my opinion.

   1. A variant of this solution would be to use GraphQL API instead of REST. And then we could use for example sullivtr/graphql to interact with it, same as with mastercard/restapi and REST. This variant was added following @pedropombeiro's suggestion from gitlab-org/gitlab!173186 (comment 2222234124).

3. Do all updates needed to add cost factor support in GitLab Provider for Terraform. Which means: extend GitLab Rest API, extend go-gitlab library and extend GitLab Provider for Terraform to pass those values, even if they are usable only for instance admins on GitLab.com and no other instance. That would be the perfect solution, but it requires updating multiple dependencies, so will be definitely the longest path.

   1. As I've just learned we can use our GraphQL API in the GitLab Provider for Terraform, a new possible variant is updating the provider to use GraphQL for the gitlab_user_runner and pass cost factor details through it. This now becomes the variant I'd propose to go with 🙂