Add GCP labels to runners

Background

I have been looking at CI minutes and how we attribute usage to internal and external users (see https://app.periscopedata.com/app/gitlab/530329/WIP:-Davis-CI-Runner-Minutes).

However, in the current state this is assuming all minutes are created equal, which is not true as runners have different instance types with differing costs.

I don't have a way currently to tie the usage I can get from ci_builds & ci_runners tables in Postgres to the machine cost I see in Bigquery from our GCE usage.

Solution

Right now all our machines in gitlab-ci GCP project are un-labeled.

I'd like for us to label these machines with 2-3 labels to address these issues.

1. runner_id

• label key: runner_id
• label values example: 6fklfnjfjs

This would allow me to join the lineitem cost in Bigquery to the jobs created by the runner with a particular ID

2. runner_manager

• label key: runner_manager
• label values example: gitlab-com-srm

I'm not sure what the best to way to specify values for this one is, but this would allow us to easily aggregate costs across each of the runner managers

3. is_runner_manager

• label key: is_runner_manager
• label values example: yes

This label would be a simple boolean to specify which costs are the runners within a particular runner manager and the runner manager itself, so we can easily aggregate costs for just the runner managers or just the runners themselves (we could probably also be clever about how we use runner_manager label to capture this also)

The best part about using a solution like this, is that we can do the same thing in other cloud providers like AWS, so it is extensible into some other execution environments

Complications

There are 2 possible complications I could see with his approach:

1. High Cardinality for runner_id label values:

• most labels are things like env:prod, env:dev and stuff like that, so I'm not sure what happens if we are throwing in runner_id:4fuhzjhff to where the cardinality in the values field grows very large.

2. cluster based execution environments like GKE:

• Can we still apply labels to the individual instances or a meaningful representation that still ties back to the ci_builds?

Links:

• GCP labels: https://cloud.google.com/resource-manager/docs/creating-managing-labels#common-uses
• Bigquery query examples with labels: https://cloud.google.com/billing/docs/how-to/bq-examples#group_by_keyvalue_pairs

Proposal

At the moment docker-machine google driver does not support labels so it's something that we have to add.

I think following the same idea of how --google-tags is the best way forward so that we can define this flag in the Runner manager config so then we just define a label for each manager. For exmpale:

[[runners]]
  name = "gitlab-runner-manager"
  url = "https://gitlab.com"
  token = "xxxxxxx"
  executor = "docker+machine"
 ......
  [runners.machine]
   .....
    MachineOptions = [
      ....
      "google-labels=runner_manager:gitlab-runner-manager-5"
    ]

So all this configuration would be handled by our chef repos. We will also have to update GitLab-Runner to that it expands the variables if it doesn't do so as well.