Skip to content

Implement Ci::RunnerMachine#heartbeat

Pedro Pombeiro (OOO from July 16th till Aug 7th) requested to merge pedropombeiro/387181/3-implement-heartbeat-on-ci_runner_machines into master

MR chain breakdown:

Sequence MR
1 Add runner_machines relationships (!109980 - merged)
2 Add logic to create ci_runner_machine based on ... (!109983 - merged)
👉 3 Implement Ci::RunnerMachine#heartbeat (!109985 - merged)

Part of Use runner token + `system_id` JSON parameters ... (#387181 - closed)

What does this MR do and why?

Describe in detail what your merge request does and why.

This MR adds a heartbeat function to Ci::RunnerMachine that is taken almost verbatim from Ci::Runner. In time, most of the heartbeat functionality will migrate from Ci::Runner to Ci::RunnerMachine.

This MR follows the Runner Token Architecture blueprint.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

The write to the database is happening every 40-55 minutes if any cached properties change:

image

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

This is how I checked the functionality locally (the instructions assume your local GDK is at http://gdk.test:3000, adapt accordingly):

  1. Create a runner with glrt token prefix. Since we don't yet have the service or GraphQL mutation that will handle this, we need to do it from the Rails console:

    creator = User.find(1) # root user
project = Project.find(20) # take the ID of a target project
runner = Ci::Runner.new(legacy_registered: false, creator: creator, runner_type: :project_type, projects: [project], tag_list: %w[gdk macos shell]).save!
puts runner.token # Use this token in the next step

  2. Register the runner with the authentication token:

    gitlab-runner register -config ~/.gitlab-runner/config.gdk.toml \
        --executor "shell" \
        --url "http://gdk.test:3000/" \
        --description "Playground project test runner" \
        --non-interactive \
        --token="glrt-..." # Use token from step 1

  3. Schedule a CI job in the project. You can use the following yaml:

    build1:
  stage: build
  script:
    - echo "Do your build here"

  4. Execute the runner:

    gitlab-runner run -config ~/.gitlab-runner/config.gdk.toml

    Notice that the deprecation warning is not shown since the token starts with glrt-.

  5. Once the runner has picked up the job, run the following query in the Rails console to verify that no ci_runner_machines record was created (since the :create_runner_machine FF is disabled):

    > runner.runner_machines.count
=> 0

  6. Enable FF:

    Feature.enable(:create_runner_machine)

  7. After a few seconds, the runner should have polled the GitLab instance for new jobs and caused the ci_runner_machines record to be created:

    > runner.reload.runner_machines
=> [#<Ci::RunnerMachine:0x0000000294e32ba0
  id: 1,
  runner_id: 1,
  executor_type: nil,
  created_at: Fri, 10 Feb 2023 13:03:34.994229000 UTC +00:00,
  updated_at: Fri, 10 Feb 2023 13:03:34.994229000 UTC +00:00,
  contacted_at: Fri, 10 Feb 2023 13:03:34.986092000 UTC +00:00,
  version: nil,
  revision: nil,
  platform: nil,
  architecture: nil,
  ip_address: nil,
  config: {},
  machine_xid: "s_c2d22f638c25">]

  8. Although some properties show as nil, their values are cached by Redis. You can confirm their presence:

    > runner.runner_machines.first.version
=> "15.9.0~beta.82.g12d9fa39"
> runner.runner_machines.first.ip_address
=> "127.0.0.1"
runner.runner_machines.first.executor_type
=> "shell"

    After 40-55 minutes they will be flushed to the database, as shown in the screenshot section.

  9. Test that old runner versions also cause heartbeats to work:

    docker run --rm -it -v $HOME/.gitlab-runner:/etc/gitlab-runner:ro --add-host=gdk.test:$(ifconfig | grep inet | awk '$1=="inet" {print $2}' | tail -n1) \
  gitlab/gitlab-runner:v15.5.0 run -config /etc/gitlab-runner/config.gdk.toml

    In the Rails console, notice the new runner machine with the <legacy> machine_xid:

    > runner.reload.runner_machines
=> [#<Ci::RunnerMachine:0x000000012be3b350
  id: 2,
  runner_id: 1,
  executor_type: nil,
  created_at: Fri, 10 Feb 2023 13:18:56.986738000 UTC +00:00,
  updated_at: Fri, 10 Feb 2023 13:18:56.986738000 UTC +00:00,
  contacted_at: Fri, 10 Feb 2023 13:18:56.960375000 UTC +00:00,
  version: nil,
  revision: nil,
  platform: nil,
  architecture: nil,
  ip_address: nil,
  config: {},
  machine_xid: "<legacy>">,
 #<Ci::RunnerMachine:0x000000012be3b1c0
  id: 1,
  runner_id: 1,
  executor_type: nil,
  created_at: Fri, 10 Feb 2023 13:03:34.994229000 UTC +00:00,
  updated_at: Fri, 10 Feb 2023 13:03:34.994229000 UTC +00:00,
  contacted_at: Fri, 10 Feb 2023 13:03:34.986092000 UTC +00:00,
  version: nil,
  revision: nil,
  platform: nil,
  architecture: nil,
  ip_address: nil,
  config: {},
  machine_xid: "s_c2d22f638c25">]

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Pedro Pombeiro (OOO from July 16th till Aug 7th)

Merge request reports